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1 • Introduction 



Purpose 

The problems investigated in this study are spacif ied within 
the framework of four questions: 

1* Whea a user of <i:n online system resd^ews the results 
of the search, is the user^s designation of th(i 
value of the search (the ability of the results to 
meet the need prompting the search request) related 
to the user's designatior of relevance, satisfaction 
with the proportion of relevant citations > concern 
for recall^ and the perception of the searcher in 
the search interview? 

2. Do specified user groups (students and faculty) 
differ in their assessment of search parameters 
and in their participation in the search interview? 

3. In the search interview, is information-giving 
activity on the part of the user related to the 
user assessment of results in the areas of rele*- 
vance and concern for recall? 

4. If user information-giving activity is related to 
user assessments of relevance and concern for re- 
call, is this relationship enhanced by question- 
asking activity by the searcher? 

Supplementary factors related to these questions are also 
considered. 

Background 

Growth in the accessing of online bibliographic data ^^ases 
appears to be a continuing phenomenon* Over a period of ''^f our years 
the nxjmber of such searches almost tripled, from 700,000 in 1974 to 
2,000,000 in 1977 (Williams, 1977). Williams considered several 
factors to be significant in supporting this growth, (1) increased 
use of the National Library of >iedicine (NLM) System; (2) the emer- 
gence, in 1977, of a new online vendor. Bibliographic Retrieval 
Services (BRS); (3) increased exposure of students to online system 
in higher education; and (4) decreasing costs per record searched. 

Interest and activity related to the provision of online search 
services Is high. Hie Reference and Adult Services Division of the 
American Library Association has formed a special discussion group 
on machine assisted reference services ("Librarians on M^S,'* 1974) ♦ 
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A coaf etence on The Online Revolution In Librarlea % scheduled by 
the University of Pittsburgh, and for which 300 attendees were 
anticipated , drew inore than 700 interested librarians and library 
educators (Nyren, 1978)* The advent, in 1977, of two new journals 
devoted to online access to bibliographic data bases, QNLItiE and 
Online Review , provides support for the belief that the growth of 
online users will be sustained « 

Since it is likely that online services will become more connon 
iti libraries and information centers in the future, it appeared 
appropriate to examine aspects related to the effectiveness of these 
services in meeting information needs* 

Within the confines of this study effectiveness measures are 
user designations in the areas of search value, relevance, satls^ 
faction with the proportion of relevant citations, concern for re- 
call and perception of the seardier in the search interview* Aspects 
specifically examined are interrelationships of user assessment 
designations, differentiation in assessment and search Interaction 
fey types of users (students and faculty) , and an lamination of 
theoretically significant roles of the user and title searcher In the 
search Interview and their sxibsequent relationship to effectiveness 
measures* Excluded from this study is any specific examination of 
actual search strategies performed for an individual search, although 
this obviously has a considerable impact on search results. However, 
the focus outlined here (user assessments and interview factors) 
appears to contribute, in part to supplying information for gaps 
suggested by Swanson (1975) vrtilch impede a better understanding 
of the complex search process* * 

Studies of personal, largely verbally supplied services 
such as answers to reference queries and seardi strategy 
assistance are scarce**^^ Could guidelines be provided 
to information personnel to maximize their performance 
in this cofflffiunicatlon process? The question cannot be 
answered without appreciable data on characteristics of 
these communications including behavioral data for both 
patrons and information personnel, (p. 148) 

Problem Rationale 

The questions outlined as the purpose of this study suggest 
that three areas are of interest, (1) the interrelationship of 
assessment measures. (2) assessment differences exhibited by 
different types of sers, and (3) the search interview* 

Interrelationships . Even thotj^h varying question formats 
are used, a number of evaluation questionnaires used by institutions 
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providing retrieval services appear to elicit evaluative infonaation 
from the search user in both specific areas (e-g,» relevance of 
citations^ absence of known relevant citations > timeliness* feelings 
about the interview* amount of time saved) , and in a more general 
overall area (e»g.» value of the search* satisfaction with the search* 
helpfulness of the search) . Four question types commonly occuring 
in these questionnaires are: of the citations retrieved how many 
did the user find to be relevant; did the user feel that the search 
missed any relevant citations; ^at did the user feel about the 
search interview; and did the user find the search results to be 
valuable (Daniels, 1978; Hitchingham* Note 1) • 

If overall evaluative assessments are considered to be rep- 
resented by questions on previous evaluations which ask the user 
to respond, on a fomr- or three-poliit scale* whether the search 
was* for example, of Major Value, Considerable Value* Minor Value* 
No Value (Lancaster* 1972; Mick* 1977); or Very Usefxil, Of Some 
Use, Of Little Use, Of No Use (Carmon, 1975); or Very Helpful, 
Helpful, Moderately Helpful, Not Helpful (Tagliacoeso, 1977); or 
Very Satisfactory* Generally Satisfactory, Not Satisfactory 
(Benenfeld, Marcus, Pensyl* and Reintjes* 1975); it is noted that 
the majority of users (60-91%) respond by checking the first two 
most favorable categories* 1!his si]«gests that users of retrieval 
systems will generally find that the results are valuable or sat*- 
is factory, whatever other assessments they make* From an inter* 
pretive and diagnostic viewpoint It is desirable to know how more 
specific user assessments relate to the overall assessment. 

User Characteristics . Some users of <»line information re-- 
trieval sys tarns approach the system with a greater background 
knowledge of the topic to be investigated in the search. This 
background knowledge is the result of a contintiing involvement with 
the specific area of investigation, i.e., the search is initiated 
to supplement already existing knowledge, of literature in the sub- 
ject area. In his review of experimei^t^l relevance studies* 
Saracevic (1970, p* 137) concludes that greater std)ject knowledge 
on the part of the assessor leads to more stringent assessments of 
relevance. In the same vein. It mi^t be escpecced that greater 
knowledge of the literature of a sxibject allows more stringent 
assessments of recall* l^e,* the knowledgeable user can note, and 
is concerned with* a lack of completeness in search results. 
Knowledgeable users may also place more stringent demands on the 
searcher in regard to their assessment of the search interview. 

Since, as previously noted, users of information systems tend 
to react favorably to questions concerning the oxmrall value of the 
search ♦ it was not believed that knowledgeability of the subject 
would provide a distinguishing factor when the overall assessment 
was considered. In a similar manner, it was not expected that 
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knowledgeability would provide dif £erentiar.lon In the aetiafactioii 
with tl e proportion of relevant citations * Tills satisfaction level 
is tnore likely to indicati^ a tolerance to sifting the wheat from the 
chaff in the retrieval results, and thus it would appear to be more 
related to individtial temperament than to knowledgeability* 

In this study faculty members who used an online service for 
research purposes, ^ere research was considered to relate to grant 
activity or the publication process > were considered to be literature 
knowledgeable users* Faculty members can specialize in a limited 
area and thus can be more familiar with that area^ It is generally 
asstnied that there is some pressure » if only in the tenure process, 
for faculty members to publish* Such pressure constitutes a job*- 
related need on the part of faculty to be literature conscious ♦ 
For example » in a study of medical school faculty publication rates 
(Pearse, Flora » Freeman, & Peeples, 1976) > the authors note a peak 
in ptAli cations per person, per year, in the early 40* s* Ihey attri- 
bute this peak to variations on thesis work followed by a decline 
until a new research area is selected* 

In contrast, most students, because of their role as students, 
have CO focus on a number of subject areas ^ Subject areas may be 
related, but there is less opportunity, because of conflicting informa- 
tion needs, to concentrate in depth on specific topics, and to have a 
cumulative record of involvement with one area* Ihus students, in 
general, were considered to be less literature knowledgeable* 

The Interview ♦ Among several themes noted in publications con*^ 
earning the library reference interview process (and by correlation 
the interview process for online searching) , an important role of the 
intermediary (searcher) as a question asker emerges ♦ For example, 
Francillon (1959) indicates that ^e first question often does 

not express the real intent of the requester. It is often necessary 
for the librarian to ask other questions'* (p, 193) • Taylor (1968) 
states, ^*Reference librarians and information specialists have 
developed, both consciously and unconsciously, rather sophisticated 
methods of interrogating users" (p. 179). In an instructional module 
for negotiating the reference query, Jahoda (1975, p. 12) indicates 
that the reference librarian should use open questions in the initial 
negotiation stages, and employ closed questions at the final scage of 
negotiation 

It follows that if question- asking is an important role for the 
searcher, then informations-giving actions are significant in the user's 
role in the Interaction process* Tessier, Atherton, and Crouch (1977) 
suggest that during the interaction for planning a computer search 
'*a user will provide an immense amount of Information about his re-- 
quirements, expectations, and compromises'* (p* 386) • 
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It was considered chat the aiaiount of information given by the 
user might impact on search results^ Since this information could 
be either voluntary or searcher elicited, it was desirable co assess 
the relationship between the searcher *s asking of questions and the 
user ^3 giving of information, and the user^s ultimate assessment of 
the results* 
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Reviev of the Literatutre 



Sources relevant to the three areas considered in this study 
(user assessments, user differences, the interview) are noted in 
this section* Assessment parameters and users are treated together 
since stxidies Intermix these aspects ♦ The interview is considered 
in a separate section. 

Assessments and Users 

Although the relevance study conducted by Rees and Schxiltz (1967) 
was not an examination of an operational system, but rather an assess- 
ment of experimental relevance judgments (i*e*, defined groups of 
individuals making relevance judgments on test documents) it is in- 
cluded in this review becaxise of some suggestive results concerning 
differences in relevance assessments by groups of people making a 
relevance decision* As the authors note, '*It was assumed that dif-^ 
ferences in the extent of sd^ject expertise and experience in research 
would result in variations in conceptualization of the information 
need as revealed by differences in the relevance ratings'' (Vol* l,p,29)» 

Subjects were 184 judges divided into five expertise^related 
groups* Hie groups were comprised of (a) 40 medical experts in the 
field of diabetes (i*e*, MD's, 14 of whom were researdxers, and 26 
who were involved in patient care or medical education) ; (b) 29 
medical scientists (individuals with a Ph^D* and working in bio-* 
medical research); (c) 25 residents (MD*s involved in clinical care 
of patients); (d) 29 second year medical stijdents, assumed to have 
limited expertise and practice in biomedical research, and (e) 61 
medical librarians with little, or no direct experience in bio- 
medical research (p* i8) • The judges evaluated 16 documents for 
their relevance to a detailed description of a research project 
concerning sugar transport in the intestine • The dependent, evalu- 
ative measures consisted of four questions directly considering 
relevance (overall relevance, relevance with respect to the formula- 
tion of the research problem, relevance with respect to the inter- 
pretation of the finds) , and one question concerning the overall 
usefulness of the document with respect to the research. 

Although the total study covered a number of aspects of the 
relevance judgment (e*g,, relevance judgments in regard to different 
kinds of document representation — citation, abstract, full text; 
and relevance judgments made in response to hypothesized stages of 
the research problem — initial formulations, carrying out the 
experiment, and analysis of results), only those findings sximmarized 
in the conclusion which are related to the current study are con-- 
sidered here* 

There were significant differences in the mean ratings assigned 
by judgment groups to the document set. Scientifically oriented 
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groups (e.g*> medical experts and medical scientists) assigned 
lower mean relev^ce ratings than did less scientifically oriented 
groups {e*g*, residents and medical librarians) when the overall 
relevance assessments are cc^sidered (p* 271) » A more refined 
breakout » analyzed earlier In the study, indicates that the actual 
ranking by group ^ of mean relevance ratings includes > from high to 
low^ medical librarians, medical experts (non-researchers), resi- 
dents, medical experts involved in research, and medical scientists 
(p. 179), Thus, mean relevance ratings appear to become more strin- 
gent as one continues along a scale from clinical to more direct 
research involvement with the problem* 

A second finding of interest to the present study was the 
indication that, at least on a document to document basis, responses 
concerning the usefulness of the document were highly correlated to 
the responses no the question concerning overall relevance (p, 273). 
The aspect of this finding that is considered pertinent to the present 
study is whether a similar high correlation between the relevance 
figure determined for a search, and the user's determination of the 
value of the search in meeting the need prompting the request, would 
occur . 

The AIM-TWX study conducted by Lancaster (1972) appears to be a 
seminal study in the evlauation of online searching because it is one 
of the first to explicate the idea that one of the best indications 
of the success of a search is the user's subjective assessment of the 
search's value. In addition, the four-point value scale (major value, 
considerable value, minor value, no value) developed by Lancaster ap- 
pears to have influenced the decision of subsequent investigators 
(e*g.^ Carmon, 1975; Tagliacozzo, 1977; Mick, 1977; Jahoda, Bayer, and 
Needham, 1973) to adopt the same scale or to adapt the scale in a 
similar manner for their own needs* 

In the AIM--TWX study 48 users conducted their own searches on the 
Abridged Index Medicus data base vriiich w^ available for online search- 
ing at that time* Users were equally divided between those «ho had 
never used the system before and those who had used it at least once. 
Precision values were determined for each search by dividing the number 
of unique relevant citations noted by the user in the search results by 
the total number of unique citations printed (precision scores equate 
to the relevance score determined in the present study) • The average 
precision figure for 45 searches was 63«1%« As Lancaster notest this 
figure is higher than the average precision (50%) achieved in 299 off- 
line searches studied in his earlier MEDLARS investigation (1968)* In 
the MEDLARS study precision figures are based on user evaluations of the 
actual documents (not just the citations) ; in the AIM*-TtfX study users 
were given copies of the attual articles if this was necessary for 
unequivocal relevance assessments* 



7 



ERIC 




Approklmacely one-third of the AIM-TWX users considered in the 
study were M.D/s (associate and assistant professors, residents, 
interns and postdoctoral fellow) ; ten users were third or fourth 
year medical students. The remaining users included research asso- 
ciates and assistants, a physical therapist, and an executive from 
a company manufacturing medical equipment. More than half of the 
searches appear to have been initiated for a research paper, review 
article or book chapter; writing a book or thesis; or preparing a grant 
application) . 

Perhaps the most interesting aspect of the AIM-TWX study is a 
consideration of the value that users attached to the search results. 
Like users in a number of studies following AIM-TWX, the group as a 
whole is perceived as attaching a high value to their search output, 
i.e., for the 38 searches evaluated on the four-point scale, 67X were 
designated as being of major or considerable value. 

Moll (1972) reports on the evalxiati'rc responses of 62 users re- 
viewing 100 AIM-TWX or MEDLIJJE searches done at the University of 
Virginia from August 1971 through January 1972 (MEDLINE searching, 
providing greater access to the biomedical literature, replaced AIM- 

searching of a more limited online data base during the period 
studied) . More than 300 searches were conducted during the study 
period, thus the return rate for questionnaires appears to have been 
approximately 30%. For 82 of the evaluated searches, users indicated 
that the citations furnished were either "most useful," or of "con- 
siderable interst." For 18 of the evaluated searches the citations 
were considered to be of "little interest" or "worthless." It is 
noted that a total of 2,082 citations were produced for the 100 evalu- 
ated searches. For 55 of the searches an estimated 50% or more of the 
citations sent were considered by users to be helpful in their work. 
It is interesting to note that 27 of the searches for which users 
estimated that 25% or fewer of the citations supplied were helpful 
wero still considered to be searches that were "most useful" or of 
"considerable interest." This suggests that, for some users, a low 
relevance score does not reflect upon the ultimate judgment of an 
overall value of the search. 

In 1974, al^Bost two years after MEDLINE services were initiated 
at the Calder Memorial Library of the University of Miami Medical 
School, users were surveyed concerning their evaluation of the service 
(McCarthy, Maccabee & Feng, 1974). Questionnaires were sent to 350 
locatable previous users. These users had initiated more than 1 , 200 
searches in the time-frame considered. Almost 50% of the questionnaires 
were returned. 

While users were not asked to respond to a direct question con- 
cerning search value, the authors assume general user satisfaction 
from the responses noted for three questions. First, even though 73% 
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of the users indicated that their search had been paid for by institu- 
tional funding » 44? of the users indicated that they would use the 
service with the same frequency if they had to pay with their own 
funds. Second, 62Z of the respondents said that they had recieved 
about the number of citations they wanted. Finally, users were asked 
to supply specific criticistas of the MEDLINE service; for almost 60% 
of the returned questionnaires no specific criticisms were noted. 

The authors sxiggest that there may have been a difference in 
satisfaction related to the type of interview conducted (phone or in 
person) , but other than one comment in this area from a user, they 
provide no data to support this speculation. 

Benenfeld. Marcus, Pensyl and Reintjes (1975, pp. 6-1 to 6-3) 
report evaluations by users of online services at M.I.T. Under the 
auspices of the Northeast Academic Science Information Center (NASIC) , 
an experimental, pilot operation of a computer-based reference search 
service was initiated at the M.I.T. libraries in Novenber 1973. The 
user population included academic users (faculty, graduate and under- 
graduate students, staff), and industrial users. In Deceinber 1974 an 
evaluation questionnaire was sent to all users of the service from 
the time of its initiation. One questionnaire was sent to each user, 
some users receiving the questionnaire may have had more than one 
search request. Almost 50% of the questionnaires were returned. 

Through NASIC, M.I.T, users had access to a range of data bases 
in science and technology, the social sciences and the himanities. 
MEDLIME services were available through the National Library of 
Medicine. Approximately one-third of the more than 300 searches con- 
ducted for the period March 1974 through February 1975 were iffiDLINE 
searches (p . G-1) . 

Responding users were, in general, considered to be satisfied 
with the service. More than 90% of the users responding on a three- 
point satisfaction scale (very satisfactory, generally satisfactory, 
not satisfactory) indicated that they were either generally satisfied 
(41%) or very satisfied (50%) . In response to a four-point-scale 
question concerning citation relevance (high relevance, moderate 
relevance, marginal relevance, no significant relevance) most users 
of the service indicated moderate (39%) to high relevance (45%) for 
the citations they received when these citations were considered in 
relation to their initial problem. 

Almost 35% of those responding did not choose to designate their 
status (e.g., faculty or student), thus it is not possible to examine 
user responses in relation to status for this study. 

Jestes (Note 2, 1974) reports on evaluations of an online search 
service directed to the CAI^ (Bibliography of Agriculture) data base. 




In 1973 a grant to support an experimental service to provide cita- 
tions from CAIN was awarded to the Library of the University of 
California at Davis, The report notes implementation procedures 
and costs as well as user evaluations collected after three months 
of service operation. 

The population served included researchers in departments and 
laboratories at the campus, and Agricultural Extension Specialists 
who attended a demonstration at the university. The service was 
free to users, but they were given an indication of the search cost 
when they received their results. Questionnaires were sent with 
each bibliography; of a total of 237 questionnaires sent, 87 (37%) 
were returned. 

Responses for three of the questions are considered here. 
Relevance of citations is reported as a total for all citations 
received by respondents completing the questionnaire. Of a total 
of 10,333 citations sent, 36% were considered to be directly per- 
tinent to the research problems of the user, 25% were thought to be 
interesting but not directly pertinent, and 37% were considered not 
useful (2% of the citations were not accounted for). Even though the 
percent of directly relevant citations was not particularly high for 
this study, users gave some indications that they were satisfied with 
the results. Almost 90% of the respondents indicated that the search 
was worth the cost ($11.61 was the average cost per bibliography), and 
67% Indicated that in the future they would be willing to pay for such 
a service. 

This report is interesting because of the results concerning 
relevance and perceived satisfaction, i.e., a group comprised, for 
the most part, of academic researchers indicated a relatively low 
overall precision score, yet appeared satisfied with the results, 
so insofar as satisfaction can be equated with a willingness to pay 
for future services. 

Hoover (1976) reports survey results from users of online services 
in another academic environment, the University of Utah libraries. In 
April-May 1975 all users received an evaluation questionnaire with their 
results. Of the 76 questionnaires distributed 26 were returned. The 
survey period occurred almost 15 months after the initiation of ser- 
vices at Utah. 

Most respondents were faculty or graduate students. Sixty-fotnr 
percent of the respondents indicated that they were first-time users. 
Multiple data bases were searched for a number of the requests. Users 
were asked to give an estimate of the percent of relevant citations in 
their results by checking a five-point-scale (e.g., 0-20%, 20-A0%, . . . 
80-100%). Fifty-four percent of those responding indicated that 60% or 
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more of the citations were relevant* All of the respondents an-^ 
9we red yes when queried whether the service was worthwhile* Hoover 
indicates that faculty groups had the highest success (i«e.» more 
faculty members indicated 60% or greater relevance) » and suggests 
that this occurred because they knew most precisely the topics they 
wanted to search. However, examination of the spread of responses 
shows that each user group, other than professional researcher, had 
low and high relevance searches of an almost eqiaal nund>er« 

Faculty and stud«:t users of MEDLIHE were surveyed in a univer- 
sity that did not have a medical school (Hitchingham, 1976) ♦ Hie 
study covered a period of four months. TVenty-one faculty and stu*- 
dents made use of the service ^ which was provided free during that 
period* Thirty-six seart^es in five subject fields were initiated* 
Uiirty-one of the seardies were for a research related purpose (on- 
going research, writing a research paper or review article) • 

Evaluation forms were sent with each of the completed searches* 
Twenty-seven (75%) of the evaluation forms were returned. All of the 
students returned forms, while only 13 out of 22 faculty searches 
were evaluated* Of the 27 searches evaluated 25 were foioid to be of 
major or considerable value* Users were asked to mark relevant cita- 
tions on the printout ♦ a precision value for each search was computed 
after examining the tnarked printouts. The average precision value for 
all searches was 53.9%* Pracision values achieved for the searches 
had no apparent relationship to the amount that the user indicated 
she/he would pay for such a service in the future (i*e., low and high 
precision values were noted in conjunction with willingness to pay 
from $2.50 to $50 for future searches). Higher mean precision values 
were noted for student searches (60%) than for faculty searches (47.3%) 
and it was suggested that this might be Indicative of less critical 
judgment on the part of students as to what is^ or is not relevant to 
their topic. 

Tagliaco220 (1977), in a follow-up study of MEDLINE users who had 
requested a search from one of seven midwestem centers during the pe- 
riod April-September 1973^ achieved one of the highest response rates 
noted in reviewing online evaluations. Almost 90% of the 1>017 ques- 
tionnaires reaching the addresses were returned. From an earlier study 
detailing the characteristics of all the MEDLINE users (Tagliacoz2o, 
1975 » p. 295) » these responses can be assumed to have come, for the nuDSt 
part^ from in-person users of MEDLINE as opposed to requests received 
by phone, teletype or mail* It also appears that responses reflect 
' both users who had one search > and users with more than one search (i.e., 
one user may have returned several evaluations) ♦ Almost half of the 
810 search initiations were first-time users of MEDLINE* 

Two questions directed to users asked that they consider the 
'^helpfulness'' of the MEDLINE search and the **usef ulness^* of search 
results. For the helpfulness aspect users were asked to indicate 
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whether the search was not helpful ^ moderately helpful, helpful^ or 
very helpful* For 895 searches evaluated in this item* 60% were con- 
sidered to be helpful* 

The second question asked the user to make a point on a line to 
Indicate the usefulness of the search results^ The line was anchored 
at the left by the words ''completely useless" and at the right by 
*Very useful," Later tihe line was divided into seven equal intervals to 
obtain scores (-3 to +3) * Tagliacozzo suggests that this question re- 
quirt d more precise judgment on the user's part than the "helpful'' 
question • For tiie 826 sear dies evaluated on diis question 69% were 
scored for points corresponding to +1 through +3* 

When the responses (N * 821) to the two questions were compared 
by means of a frequency matrix of helpfulness responses versus line 
scores » the author notes that there were some ambiguities in the mod* 
erately helpful and helpful categories^ For example > six responses 
indicate that the MEDLINE search was helpful, but a -1 to -3 score is 
noted for these same searches on the usefulness dimension^ Response 
patterns noted for "moderately helpful" show an amost normal distribu-- 
tion when plotted against usefulness scores* Tagliacozzo suggests 
two interpretations of this occurrence, either the word moderately 
implied positive connotations to some users, and negative to others; 
or that users marking "moderately helpful" were "those who did not 
have or were xmwilling to express — a strong opinion on the 
MEDLm service" (p. 246). 

In addition to the helpfulness^usefulness comparison, the author 
contrasts the users' indications of the number of useful references 
retrieved with values noted on the helpfulness din^nsion (N « 706) , 
Dichotomizing the helpfulness responses (higgler » very helpful, helpful, 
lower « moderately helpful, not helpful) and the number of useful ref*^ 
erences indicated by the user (0^5, and 6 or more), she notes a strong 
association between the number of useful references retrieved and the 
judgment of helpfulness ♦ More than 80% of the users with the higher 
numbers of relevant references also indicate that the search was helpfi-1 
or very helpful; only 40% of those retrieving five or fewer references* 
gave such indications. 

An assessment aspect that might be considered to relate to a 
concern for recall by the user was also studied^ For this contrast 

» 795) frequencies of dichotomized helpfulness values were related 
to the user responses indicating either that no relevant ciations were 
missed, or that one or more relevant citations were known to be missing* 
She notes a significant association in these measures, i»e., 70% of the 
responses indicating \o missed citations were aligned with upper values 
on the helpfulness sc^le; only 49% of the users noting one or more missed 
citations chose the higher categories. 
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The author notes that users may give contradictory and disso-- 
nant responses when considering several aspects of an evaluative 
questiotmaire. Furthermore, she suggests caution In interpreting 
from single responses, particularly those which elicit general or 
overall judgments that the user's information needs were, or were not 
satisfied by the service ♦ 

Relatively few evalxiation studies have been reported for users 
in a non-academic environment, Fosdlck (1977) reports the responses 
of online users at the U*S. Army Construction Engineering Laboratory 
(CEKL) in Champaign , Illinois • Evaluation forms were sett to 27 users 
noted as recipients of online searches over a two-year period; 26 of 
the forms were returned* Fosdlck used the same scale for relevance 
as that previously noted in Hoover's study (1976), Unlike the aca-- 
demic users ^ respondents at CESL indicated generally lower estimates 
of citation relevance. Almost 70% of the users indicated that 40% 
or fewer of the citations were relevant. However, in response to a 
broader question concerning the usefulness of the service ^ CERL users 
evidenced the same favorable trend apparent in many other online 
evaluations. Twenty-five of the respondents indicated that they felt 
that the service was a useful addition to other library services; 
twenty users believed that the search was worth the cost, 

A second study focusing oi\ non-academic online searching is the 
evaluative component of the final report on the Lockheed Public Library 
Experiment (Mick, 1977) , The project involved development and iin)le- 
mentation of a program to assess the viability of providing access to 
major online bibliographic data bases in the public library setting. 
Only those evaluative aspects related to the current study are con*- 
sidered here. 

During the first year of the project services were provided free 
to users of four California public libraries. In the second year one- 
half of the actual costs (terminal connnect time and printing costs) 
were charged back tc the user, IVo libraries continued with the project 
in the third year. During this period the fee system was based upon 
total recovery of connect and printing costs • 

In the first year there were 1,236 patrons who requested a search* 
For the second year (recovery of half the costs) usage dropped; in this 
period there were 611 users* In the third year (full cost recovery > 
only two libraries participating) 326 users requested an online seardi* 
Students (graduate and undergraduate) » educators ( teachers ^ professors , 
school administrators) ^d scientists /research personnel accounted for 
the bulk of users over the diree-year project* Use by graduate students 
and educators increased over the time span encompassed by the project » 
while use by undergraduate students and scientists declined as half » 
and full cost fees were introduced. 
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Although the details of eliciting evaluative information from 
users are not explained^ it appears that a questionnaire was sent 
to users at some time during the study. Response rates appear to 
have been fairly low, i^e,, evaluative responses noted respresent 
36X of the first year u^ars, 26X of the second year users, and 29Z 
of the third year users. Responses to questions concerning the search 
value ^ the adequacy of results, and the nxnii>er of citations consider<=^I 
useful are discussed here» 

In general, responding xisers showed little change over the three- 
year period in their perception of the value of the search* When 
asked to make a value designation on a four point scale (major value ^ 
considerable value, minor value, or no value), SSX to 76Z of the 
users over the three years Indicated that the search was of major or 
considerable value* 

In response to a yes/no choice concerning whether the results 
adeqtiately answered the question, 53% to 56% of the users responded 
In a positive mode. Relevance determinations for citations were not 
given, however, in reviewing the results of Mickys study it is possible 
to speculate that many users were satisfied with a fairly low proportion 
of useful articles (e*g*, almost 75% of tiie third year users found 0 to 
15 citations to be useful)* An earlier, interim report of the public 
library experiment (Summit & Firschein, 1976, pp. 3-3) indicates that 
in the second year of the study an average of 88 citations per search 
were printed offline. Mick reports that 69% of the second year users 
had 0 to 15 useful citations* 

the results from this study suggest tdiat whatever the other factors 
involved, users will give rather favorable responses to an overall ques- 
tion concerning the value of the search • 

One of the few sit^lations in which online users in two different 
types of organizations were studied has been reported by Jahoda, Bayer, 
and Needham (1978)* Tlie results presented were obtained from records 
maintained for each search, and user feedback obtained for loost of the 
searches • Participants in the study included 50 individuals from t^.e 
Chemistry Department at Florida State University (faculty, graduate 
, students at the dissertation stage, and research associates), and 234 
scientists and technologists at the Monsanto Textile Consp^tiy, Services 
were free to users dmring the 13 month period reported. Results from 
353 FSU searches and 345 Monsanto searches are rep'^rted for the free 
period. Findings related to the present study are examined here. 

Results from the two groups were contrasted. Academic users were 
found to be more interested in an exhaustive approach to the literature 
(everything available) than were the industrial users. Monsanto users 
were more likely to approach the service because they were looking for 



14 



er|c 



1 



specific faces and procedures. When the user analysis of results is 
considered, the authors note that both FSO and Monsanto users were 
generally pleased. Users were asked whether the nunfcer of citations 
they received was "about right," "too many" or "not enough." For 68% 
of the FSU searches and 64% of the Monsanto searcies, users indicated 
that they had received about the right number. He average or median 
number of total citations retrieved for the searches is not reported, 
although it is known that for the FSU searches only 25% resulted in 
more than 21 online citations being printed, and there were offline 
prints for 36% of the searches. Sixty-three percent of the Monsanto 
searches had offline citations. It seems that even though FSU users 
are characterized as being more exhaustive in their approach to the 
literature, they may have received search results which, on the 
whole, included fewer citations than those received by Monsanto users. 
There was little concern by either group that too many citations had 
been retrieved (7% of the searches at FSU, 5% at Monsanto); some con- 
cern that not enough citations were presented was more likely to be 
expressed (13% at FSU, 19% at Monsanto). 

There were no significant differences in the two groups when they 
reported their perception of the overall utility of the searches. The 
theme of generally favorable responses to an overall evaluative ques- 
tion noted previously in this review, is repeated for both user groups. 
Users were asked to indicate whether the searches were "very useful," 
"of some use," "of marginal use," or "of no use." FSU users found that 
78% of the searches were either very useful or of some use, at Monsanto 
75% of the searches were in these categories. 

Student users of online services were the focus of interest in a 
study at the University of Delaware (Kobelski & Truufcore, 1978). The 
gioups studied were satisfied and non-satisfied student users who had 
received a cost-subsidized search during the 1976-77 academic year. 
The period covered was the second year of operation for the search 
program. The subsidization system meant that students were charged 
half the normal diarges for online searching (actual connect costs 
plus 15% surcharge for paper, phone, etc.) The average cost to sub- 
sidized users was $10.31. 

One hundred and fifty-four subsidized searches were performed 
during the year; evaluation forms were returned for 107 searches (69%), 
As is true for a number of other studies the independence of responses 
is not reported, i.e., it is unclear whether each search evaluated rep- 
resents a unique user, or whether some users had multiple searches and 
thus returned more than one evaluation. Searches were run on a range 
of data bases, with heaviest use occurring for Psychological Abstracts 
Biological Abstracts and ERIC. An average of 1.4 data bases were 
searched for each request. 
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The details of the evaluation form are not reported but it 
appears that a dichotooous choice* satisfiedMot^satisf ied, was one 
of the items » since other search parameters are related to this aspect 
in the report. More than 80X of the searches evaluated were considered 
to be satisfactory. For 701 of the evaluation forms it ras noted that 
the responding user indicated that she/he would do another compute 
search in the future. 

The authors indicate that the eval\iation forma were eacamined 
for factors that would account for student satisfaction or dissat- 
isfaction. They conclude that neither the student *3 statxis nor 
purpose in requesting the search were related to student satisfaction. 
They also indicate no relationship between the cost of the search and 
expressed satisfaction. It is further suggested that no relationship 
exists between the percentage of relevant citations retrieved and 
satisfaction, because some searches in ^ich all citations were 
relevant did not satisfy the user, while other users were satisfied 
with less than one relevant citation in ten in their results. Using 
chi square they indicate a relationship between the number of cita- 
tions generated and student satisfaction. All students receiving more 
than 100 offline citations were satisfied. All students receiving at 
least 40 relevant ciations were satisfied. 

The Search Interview 

Quantitative studies related to the « earch interview are far 
less cotmi.n than those which consider user evaluations of online 
searching. Carnwn (1975, p. 6) reviewed the published literature for 
reference services and conq)uter-based retrieval services, and con- 
cluded that the what of library reference work (and by equivalence 
data base searching) was somewhat defined, but that the how aspect 
was largely undefined. Taylor (1968, p. 180) called the negotiation 
aspect of a reference interview a coo^lex act of himan communication » 
with the user trying to describe for another individual not what he 
knows, but something he does not know. At that time he suggested that 
quantitative information about the process was non-existent. More 
recently. Lynch (1977, p. 11) has confirmed that there has been little 
empirical analysis of the reference interview. Carmen's study (1975) 
is most related to the current investigation. Other reports, peri- 
pherally related, are also discussed in this section. 

Taylor's five filter model (1968) of the reference interview 
was first proposed in the late 60'3. He conducted a number of inter- 
vitaws with special librarians and information specialists to determine 
their --ethods of question negotiation. He states that this group was 
selected because they are usually concerned with substantive questions, 
receive inquiries from motivated and critical users with pre-lmowledge 
of what is acceptable as an answer, and are familiar with negotiation 
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techniques. Based upon information obtained in these interviews 
he sviggests five steps in negotiation, from which the librarian 
selects data useful in conducting the search. The five steps 
include: 

1. determination of [thel subject; ^f the requestj 

2. objectives and motivation; £of the user} 

3. personal characteristics of fthe) inquirer; 

4. relationship of inquiry description to file 
organization; 

5» ^ticipated or acceptable answers, (p. 183) 

Each step implies that the librarian will assume a question-asking 
role and that the user's role will focus on information- giving. 

Videotape scripts illustrating "poor" techniques of query- 
negotiation and elements of a "well-negotiated" query are included 
in Jahoda's instructional module for answering reference questions 
(1977 » Appendix D) . The situation simulates reference desk encoun- 
ters in an academic library. The poorly negotiated information re- 
quest involves nine questions asked by the libr-rian. In the well- 
negotiated request the librarian asks 15 questions. 

Somerville (1978) takes a more prescriptive approach to the 
search interview by including a list of 20 things the search inter- 
mediary should do during the interview. She suggests that the 
searcher: 

- Ask questions of the user to ensure your understanding 
of the subject . . . 

- Determine if the user prefers a comprehensive search or 
a narrow one . . . 

- Make sure that you identify all the restrictions that the 
user wants placed on the search strategy, (p. 23) 

Sotae correspc-»dence with the five filter Hiodel is evident. 

Implications and prescriptions for query negotiation have gone 
beyond the purely mechanistic aspects of the interview. They often 
include not only what should occur (the transfer of information) but 
also the manner in which it should occur. For example, Jahoda's list 
of good and poor ways to negotiate a request suggests that in the 
good negotiation the librarian will make eye contact with the user, 
give the user full attention, make the user feel at ease, show 
empathy for the user, and be aware of non-verbal cues (1977, p. 25). 
Somerville 's list of DO's" notes that the interviewer should make the 
user comfortable in disnnssing information needs by utilizing inter- 
viewirjg and counseling techniques (p. 23). 
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Gothberg (1975, 1976) addr-isseO the non-mechanistic aspect of 
the reference interview in her study of the effect of imnftdiate and 
nonlnunediate verbal-nonverbal conmnmicatlon behavior by librarians 
on tiser satisfaction. TWo reference librarians were trained to 
display iimediate or nonimnediate verbal-nonverbal comnunication 
during a reference Interview in a public library. Immediacy indicates 
a quality of liking or closeness in an Interpersonal relationship. 
Each librarian was involved In inmediate and nonimaadiate reference 
interviews. Reference transactions were recorded and video-taped. 
Library users (60) vere selected on the basis of their availability 
in the library, need to negotiate the reference question and willing- 
ness to answer the questionnaires. 

After the interview the Investigator approached the user and 
asked her/him to indicate » by means of a questionnaire, satisfaction 
with the reference interview, satisfaction with the user^s own per- 
formance in negotiating the reference question, and satisfaction with 
the transfer of information. Verbal and non-verbal components of the 
recorded interviews were coded for immediacy and noniirmediacy . 

User satisfaction with the reference interview and their own 
performance in participating in the reference Interview were signi- 
ficant for interviews with inaaediate connMmlcatlon. However, when 
the user's satisfaction with the transfer of Information was con- 
sidered there was no significant effect of immediacy in interviews 
and this satisfaction measure^ The author observes that lack of 
effect in this area may have meant that displays of verbal and non- 
verbal liking were not sufficient to bring about the trust necessary 
for users to divulge their true needs and lack of knowledge about 
the library and its tools. 

Gothberg 's study is interesting for several reasons. First, 
it appears that some defined aspects of user satisfaction may relate 
to the socio-behavioral qualities of the reference interview, while 
another measure (satisfaction with the transfer of Information) may 
not. Second, the study Is of methodological Interest because it is 
one of the few studies isolated which incorporates observations of 
a real interview situation, quantifies the observations, and assesses 
the relationship that exists between the quantified observations and 
a user satisfaction measure. Finally, in its focus on the social- 
emotional aspects of the Interview, it provides a direct contrast to 
the present study which primarily focuses on a task area of the inter- 
view (inforoatlon- giving and question-asking). 

Lynch' 3 examination (1977, 1978) of reference interviews in 
public libraries is more like the c».r?rent study of the search intar- 
vlew for it also focused ou the mechanistic aspects of the interview. 
However, user satisfaction was not considered in the study. 
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Lynch posed eight questions: 



1. How often does a reference librarian interview 
the patron who presents a reference query? 

2» Does this frequency vary acc-^rding to the type 
of transaction involved? 

3. Are interviews more frequent when the librarian 
is less busy? 

4» When an interview does occur, what gross cate- 
gories or levels of information are sought by 
the librarian? 

5. How often are the questions of the librarian open 
questions and how often are they close 1 questions? 

6. Does the reference librarian use the seconda:ry 
questions (probes) used by other interviewers? 

7. How does a librarian discover that the que.cy a 
patron first prt.-'ented is not the query he/she 
wants answered? 

3. How many primary quescions does the librarian 
ask the patron in an interview? (1977, P. 43) 

Reference interviews in four public libraries were recorded. 
Of the 366 interviews recorded, 309 were ultimately transcribed. 
Tapes were not transcribed in their entirety. Since question- 
asking by the librarian was the focus of the study, only questions 
that the librarian asked were transcribed, along with enough addi- 
tional material to make them meaningful. Transactions were clas- 
sified according to their nature, i.e., directional, holdings 
transactions, substantive or moving. Interviews were considered to 
occur in those cases in which the librarian asked the patron one or 
more questions. The investigator developed an 11 category scheme 
for analyzing holdings* transactions, and a 20 category scheme for 
substantive transactions. In addition, questions were analyzed on 
an open/closed scale, and a scale to assess probing questions by 
the interviewer. Some difficulties in coding were noted. Findings 
related to questions five, six and eight posed by the investigator 
are considered here. 

Open questions, I.e., those which allow flexibility in user 
response, were employed infrequently in the interviews. In holdings 
and substantive interviews they comprise eight precent of the ques- 
tions asked. Ninety percent of the questions were closed. T!wo 
percent were considered to fall into an Intermediate category. 
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The author notes that the sixth question concerning secondary 
or probing question^ by the librarian could not be resolved (1978, 
p. 32). In coding there was difficulty in establishing whether a 
true probe occurred (a librarian tries to elicit more fully infor- 
mation about an aspect of the patron's information need), or whether 
the question by the librarian was acttially an attempt to be sure that 
the patron's original question had been heard correctly. 

Primary questions (questions by which the librarian introduces 
some aspect of the patron *s search for information and asks for con- 
tent new to the interview) were infrequently used in the interviews. 
Fifty-two percent of the interviews involved only one question. 
Eighty-nine percent of the interviews had three or fewer questions. 

The final study reviewed here (Carmon, 1975) has had a direct 
influence on some of the aspects of the present investigation. 
Carmon 's study had two goals; first, to collect descriptive and 
quantitative data concerning the reference process tor computer- 
based literature searching, and second, to develop a modej. or the 
user interface for a projected network model of system use. 

The study was conducted at two sites, the University of Georgia 
and UCLA. Each site provided batch-mode searching of a number of 
data bases. Georgia's system allowed for both current awareness 
searching (SDI) and also retrospective searching of earlier years 
on the data bases. UCLA, provided current awareness searching only. 
Evaluative components in the Georgia results do not make a distinc- 
tion as to whether retrospective or current awareness searches are 
being considered. 

At the University of Georgia interv-t js were conducted by four 
reference specialists; these specialists were by job function in- 
volved almost eaorlusively with coii5>uterized literature searching. 
i:k:LA raaults originate form searches profiled at a nuTrf>er of in- 
stitutions in the California system, with the actual con^uter run 
being accomplished at UCLA. Searchers in the California system 
were more likely to have several other job responsibilities in ad- 
dition to computerized literature searching. 

The University of Georgia results cover a period of five months 
during which 333 users initiated a search request. Approximately 
two-thirds of the users were graduate students, although some over- 
lap in status categories is noted. At UCLA graduate students com- 
prised almost half the population studied. TWenty-five percent of . 
the interviews at Georgia were recorded, and ultimately 44 of these 
interviews were transcribed. During die same period UCLA processed 
almost 150 seardi requests. Fifty-nine of the UCLA interviews were 
recorded and transcribed. 
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User evaluation forms were sent to University of Georgia and 
UCLA users. Twenty-five percent of the total University of Georgia 
users returned an evaluation form. For searches with recorded in- 
terviews there was a return rate of 77%. At UCLA 52 user evalu- 
ation forms were returned. 

On a general satisfaction measure (very useful, of some use, of 
little use, of no use) responding users at both sites indicated that 
they were satisfied with the results. At Georgia 96% of the searches 
that were taped were found to be veiy useful, or of some use; for all 
the returned evaluation forms 89% of the searches clustered in these 
categories. At UCtA 91% of the returned evaluation forms indicated 
that the user found the search to be either of some use or very usefiil. 
At both sites the most frequently ranked specific dislike expressed 
about the service concerned recall, i.e., users ranked no way to 
judge completeness high on the disliked features list. 

Transcribed interviews were coded at each institution. Slightly 
different, self-developed coding systems were used at each site. 
Frequencies for events were noted by the occurrence or non-occurence 
of specific events; these events were aot ultimately related to the 
users expressed satisfaction measures. For both sites it was deter- 
mined that the events coded were likely to occur throiighout the 
interview. For example, the searcher might focus first on question 
negotiation, inject some descriptive information about the system, do 
some strategy development, and return to more question negotiation. 

The low frequencies noted foranumb-er of categories in the trans- 
cribed interviews were of interest. For example, at Georgia there 
were no instances in which the searcher asked the user to restate the 
question, an occurrence which might elicit additional information. 
Asking the user whether any other concepts should be added to the 
search occurred in only 10 of the AA transcribed University of Georgia 
interviews. The users were asked to give an estimate of the available 
published literature in nine interviews; relevant author terms were 
sought from the user in thirteen of the interviews. Language require- 
ments of the user were directly sou^t in ten interviews; the user was 
asked to confirm the strategy in only diree ot the interviews. If, 
as suggested in previous writings on the interview process, question- 
asking is an important role for an interviewer, it seemed that an 
examination of the magnitude of question-asking as related to user 
information-giving and the ultimate assessment of results would pro- 
vide an interestii^ framework for teaching the interview process , and 
O.1S0 practical application in actual search interview situations. 
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III. Hftthods and Proc^dur^s 

This section considers methodological and procedural aspects 
of the study which establish the perspective for reviewing results. 
Specifically discussed are (e) the research design, (b) limitation 
of the study to a particular data base, (c) the environmental con- 
straints which constitute the framework from which subjects were 
selected, (d) data collection inatrtjments employed in the study, 
(e) data collection procedures, and (f) data treatment methods 
utilized in generatijig measures appropriate for describing the re- 
sults . 

Design 

The study is characterized as descriptive research focusing on 
three areas. These include the ejcamlnatlon of interrelationships 
of user assessment areas often included in evaluating schemes applied 
to oper;itional online search services; the examination of differences 
in user response styles exhibited by defined subsets of the universe 
of users who utilize online search services; and the examination of 
relationships between specified events occurring In the search inter- 
view and the user assessment of search results. 

Data were collected In a natural setting, that is, collection was 
Incorporated Into the normal pattern of procedures enqjloyed in respond- 
ing to a request by a user for an online search. Artificial aspects 
introduced in this study Included asking the user to complete a request 
form, tape-recording of the search interview, and requesting that cer- 
tain users participate in an evaluation of search results (question- 
naires mailed to users) . 

Interviews were transcribed, and quantified by means of a 
particular content analysis scheme (Bales, 1950) applicable to the 
questions examined in this study. 

Online Data Base 

This study was limited to interviews and searches conducted 
for a particular online data base, MEDLINE. Several factors sup- 
ported this choice. First, the MEDLINE data base has been widely 
available for searching for a relatively long time (McCam & Lelter, 
1973) in contrast to some of the other online data bases available 
through the comnerclal vendors. Because of this, it offered the 
opportunity for investigation in an atmosphere uncanp Heated by 
variables associated with the use of more recently available data 
bases (e.g., few users avails le for study because they are as yet 
unaware of a new service; searchers interviewing for, and searching 
on a data base which is unfamiliar to them) . Second, because of its 
broad subject content MEDLINE is often applicable to requests in a 
nun4>er of interdisciplinary areas (medicine, biology, psychology, 
and some social sciences) . Interviews for MEDLINE may mirror to some 
extent the interview employed in searching other data bases. 
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Third, certain aspects of question negotiation for MEDLINE searching 
(thesaurus use» strategy development, establishment of preferences 
for tiiue coverage and citation format, etc.) were likely to provide 
a common denominator in intervio/s at all of the search sites; further- 
more, these aspects are comparable to those utilized in question 
negotiation for several other data bases. Finally, an earlier ex- 
perimental study of relevance judgments focused on assessments made 
by groups of individuals in the biomedical field (Rees & Schultz, 1968) 
and established, within this context, a basis for believing that depth 
of research knowledgeabllity may be associated with more stringent 
relevance assessments. An examination of this aspect in an operational 
situation was of interest in the current study. 

Thuis, the limitation to an examination of MEDLINE searches is 
not to be viewed as a specific attempt to evaluate the MEDLINE system, 
but rather MEDLINE was chosen for study because it is a data base which 
has many features common to other data bases, and MEDLINE users were 
likely to be similar to users examined in an earlier study. 

Sampling Plan 

Subjects were In-person users and searchers at three institutions 
providing MEDLINE searches. The following sections detail character- 
istics of sites, searchers, and users which define the sampling frame- 
work. 

Sitas . Collection sites included the Vera Sliiffman Medical 
Library at Wayne State University in Detroit; the Medical Center 
Library at the University of Michigan in Ann Arbor; and tiie Houstcai 
Academy of Medicine-Texas Medical Center Library. The first two 
libraries were included as participants because they had e3q>ressed 
interest in cooperating in the study, and their locale was convenient 
for regular visits by the investigator to pick up tapes and other 
materials. The third library was included when data collection 
appeared to be progressing at a rate slower than anticipated. Ma- 
terials from Houston were mailed to the investigator. All insti- 
tutions, searchers and users were promised anonymity in the reporting 
of final results. 

The Shiffman Medical Library serves both the immediate popula- 
tion of faculty, students and researchers associated with Wayne State 
University, and a more diverse group of urban and hospital-related 
vtsers with information needs related to the health care field. 
Shiffman is also the Regional Resource Library for Kentucky, Ctiio and 
Michigan (KOM) , 

The Medical Center Library at the University of Michigan is 
located in the hospital complex of the University Medical Center. 
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The Library serves the Inforaation needs of researchers, faculty, 
students, and cllQlcians affiliated vlth the Hedlcal School and 
other health profession schools of the University. It also serves 
as a resource for Individuals associated with a nxmber of prondnent 
research institutes at the Uni'^mrslty. 

the Houston Acadea^r of Hedlclne-Texas Medical Center library 
is a consortium library governed by representatives from major 
participating institutions — the Houston Academy of Medicine, 
Baylor College of Medicine, Texas Women* s TTniverslty, the Univer* 
sity of Texas at Houston, and Texas Hedlcal Center, Inc. In addition, 
it is the Regional Resource Library for the South Central Regional 
Medical Library Program. 

All of the libraries charge for MEDLINE searches, witii basic 
fees ranging from $5 to $15 or more, depending on the number of 
backflles searched and the format of the printouts (e.g., inclusion 
of abstracts) . One library prefers to have the user present for the 
interview and terminal search, another library intermixes tiiese pro- 
cedures (some interviews are confined with the search, some sear dies 
^e mn after the t^er leaves), and the third library, in most cases, 
separates the two procedures (the search is run sometime after the 
interview with the user) . However, some terminal searches with the 
user present were noted as occurring in all of the libraries for this 
particular study. 

Searchers . Eleven searchers participated in the study. All 
searchers can be considered to be relatively fandliar with MEDLINE 
since they had conducted 125 or more searches in the 12-month period 
prior to the study. One seardier had at least a year's experience 
with MEDLINE; all others had two or more years of experience. Under- 
graduate majors of searchers were predominaatly in areas othef t^an 
the physical or life sciences. Two searchers had undergraduate 
degrees in the sciences. 

Searchers indicated that training for MEDtlNE searching had 
been accomplished in a variety of ways. Four searchers had at- 
tended training sessions of varying lengths at the National Library 
of Medicine; the others were trained for MEDLINE searching through 
several uechanlsms (singly or in combination) . These included NLM 
sessions scheduled in their vicinity, use of the MEDLEAEN training 
sequence, training by a more experienced colleague, and self- teaching. 

Users . Several limiting factors, either extem:il to the study 
or design imposed, determined the sample framework of the potential 
pool of users Investigated in this study. 
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The pool from which users could be drawn Included all unique, 
In-peraon users requesting a MEDLINE search at one of the search 
sites during the data collectiai period. Thus excluded from the 
outset were search requests received by mail or phone, search re- 
quests from a user vixo had already participated in an earlier 
session, and search requests from users who asked for another data 
base even though MEDLKIE may have been subsequently used to satisfy 
the information need (all of the sites provided a range of search 
services on data bases other than MEDLINE) . 

A second limiting factor on the potential pool of users was 
self-selection of individuals for participation in the study. In 
recognition of concerns for user and searcher privacy In regard 
to tape-recording of interviews, and in compliance with guidelines 
from several cot mittees charged with monitoring all studies which 
Involve human experimentation, a consent form for participants was 
developed (Appendi.. A) » Searchers were asked to note the reason for 
not recording all searches they conducted during the data collection 
period. Known decisions not to participate (either searcher or user) 
occurred in a ratio of le*«s than 1:2 when considered against decisions 
to participate. The ratio may be higher since less than complete 
information was available -rOTj one data site. In several instances 
searchers noted that the user 'lid not wish to participate because the 
user was "in a hurry" and did not wish to take the time to complete 
the request form. In almost half the non-participating occurrences 
Che decision not to record was searcher initiated and appeared, in a 
number of instances, to be related to lack of immediate access to the 
tape recorder or general lack of time to complete all the procedures. 
Most instances of lack of tide for procedures occurred at one site which 
had a relatively high volume of searches during the data collection 
period . 

A third, design- imposed factor was the ultimate selection of 
participants to receive evaluation forms. Subjects receiving evalu- 
ation forms were selected from the total group of recorded participants 
according to design criteria, i.e., subjects selected had indicated 
that their primary status was either faculty (instructor, assistant 
professor, associate professor, professor) or student (undergraduate, 
professional school student, or graduate student). Furthermore, sub- 
jects indicating either of the previous status conditions had to have 
also indicated that the search was primarily for their own use. This 
was to preclude a situation in which the ultimate evaluator of the 
search was someone other than the person taking part in the interview. 

Faculty subjects selected to receive evaluation forms were further 
restricted in that their stated purpose in obtaining the search had to 
be research-related (grant project, preparing an article or review for 
publication, on-going research leading to publication) . This restric- 
tion was imposed to parallel the conditions noted by Rees and Schultz 
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(1967) ia their experimental study of relevance In wfeich It was 
noted that knowledgeability and Involvement in research were re-^ 
lated to more stringent relevance asseasinenta • In other words » it 
was not desired that faculty searches of a more sporadic or super- 
ficial nature (e«g«t search for Imaedlate clinical applicatioUt 
background material for a speedi» material for class instruction) 
be Included in the evaluation* 

A final situational restriction was imposed af ter-the^fact^ 
In a few cases two users participated in the interview « Since the 
search interview is considered to be primarily a dyadic coimunlca-^ 
tion process ^ these interviews would be misleadlt^ if analyzed for 
Information^giving behavior by the user* Furthermore, they would 
present problems in determining the individual most appropriate for 
evaltiating the searches • Searches of this type were excluded ♦ » 

Data Collection Instruments 

Data collecting instruments for this study included a user re-- 
quest form for collecting background Information concerning the user, 
tape recordings of Interviews , user evaluption forms sent to parti-* 
cipants meeting study criteria, and coded transcripts of selected 
interviews , 

User Request Form * T^e user request form (Appendix B) was 
developed for two purposes* First, to obtain primary information 
about each user vrtiich allowed a decision to include or exclude a 
particular user from the evaluation process* Questions 7, 10, and 
11 (ultimate user of the search, statxis of requester, and purpose 
of the search) fulfill this function* Second, for con9)arative pur-- 
poses it was also desirable to more fully describe background char- 
acteristics of the user that mi^t clarify the framework from within 
which the user made an evaluative decision* Questions 6, 12, 13 and 
15 (number of previous online interviews, degrees cotapleted, famil- 
iarity with and sustaining interest in the search topic, and ability 
to list relevant citations) were of this type* Question 14 (narrative 
search statement) was included for comparison with an earlier study 
(Carmon^ 1975) in which it was noted that users supply fairly brief 
statements concerning their search topic* Otiier questions on the 
form were designed to accommodate information usually Included by 
one or all of the search sites in their already existing forms* 

Tape Recordings * Tape recordings were utilized to capture all 
verbal interaction between searchers and users* They were turned 
on as soon as the consent fom was signed and turned off at the 
completion of the interview* 

User Evaluation Form * A standardized, generally accepted 
instrument for eliciting user evaluations of searches does not exist* 
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Hie fortn used in this study is self-^developed^ but it reflects close- 
ly areas considered in other evaluation schemes (Daniels , 1973; 
Hitchin^am, Note 1) • 

Specifically^ the evaluation form (Appendix C) was designed to 
elicit information from the user in five areas: (a) the relevance 
score for the search » ^ere relevance is determined as the nuniber of 
relevant citations indicated by the user xrtien she/he examines the 
search results, divided by the number of citations retrieved; (b) 
the user^s satisfaction with the proportion of relevant citations 
retrieved; (c) the user*3 concern for recall in considering the 
search results; (d) the user's assessment of the value of the 
search in meeting the need pron^ting thB search request; and (e) 
the user^s perception of the searcher in the interview process « 

In cases where the loser estimated relevance, the upper level 
score of the estimate was used (e^g«^ if 51% to 60% was marked, 
60% is the value) * Concern for recall and user perception of the 
searcher in the interview are summatlve measures. 

The concern for recall score is the average (mean) score 
obtained by summing user responses to the four statements relating 
to recall on the user evaluation form, and dividing this sum by the 
nuiii>er of statements responded to (three or more)* The statements 
assess whether the user (a) believes that the search retrieved most 
of the relevant citations in the data base, (b) is concerned because 
there is no way to judge completeness, (c) is concerned because the 
results omitted relevant citations Poiown to the user prior to the 
search, and (d) considers that fewer relevant citations than expected 
were retrieved* Scale values noted by the user for the second^ third, 
and fourth statements were subtracted from eleven to conform to the 
order of the first statement* Lower scores indicate more ccmcem for 
recall, higher scores indicate less concern for recall* 

The user perception of the searcher in the interview is the 
average (mean) score obtained by sixmming user responses to five 
statements concerning the searcher on the user evaluation form^ and 
dividing the sum by the number of statements responded to (four or 
more) • The statements assess, on a disagree (one) to agree (ten) 
scale, whether the tiser (a) believes that the searcher was knowl-- 
edgeable concerning the use of the data base for the question, (b) 
feels that the searcher understood the request, (c) believes that 
the searcher understood the user^s purpose in initiating the request, 
(d) believes that the searcher was thorough in exploring all aspects 
o£ the search question, and (e) believes that the searcher suggested 
terms appropriate to the subject of the request* Lower scores indi- 
cate lesser agreement to the statements concerning the interview. 

The instrument can be considered to be valid insofar as it 
reflects cotmon elements noted in other evaluations forms. Segments 
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of the form which employ auinnative measures (concern for recall, 
perception of the interview) appear to be sufficiently reliable 
when item consistency is considered. Reliability coefficients 
(alpha) of .76 for the concern for recall, and .90 for the 
perception of the interview were established (Nunnaly, 1967, p. 196). 

The major departure from most of the other evaluation forms 
reviewed was the use of a 10-point rating scheme for most of the 
questions instead of the three or four-point scale more conmonly 
used. Application of expanded scales is consistent with general 
scaling theory (Nunnaly, 1967, p. 521). 

Coded Transcripts. Transcripts of selected interviews were 
unitized by the investigator and coded by trained judges. The cate- 
gories applied to the transcripts were those defined by Bales in 
Interacti on Process Analysis (1950), The scheme is outlined in 
Figure 1. 

Interaction Process Analysis (IPA) is "an observational 
method for the study of social and emotional behavior of individuals 
m small groups (Bales, 1968, p. 465). The method centers on the 
coding of interaction miits (verbal and non-verbal) occurring when 
small groups are involved in a problem-solving task. 

An interaction unit is "the smallest discriminable segment of 
verbal or non-verbal behavior to which the observer, using the 
present set of categories after appropriate training, can assign a 
classification" (Bales, 1950, p. 37). In this study the interaction 
unit is defined as the smallest discriminable segment of verbal 
behavior. 

Units are coded according to the 12 category system. The 
first three categories are considered to represent positive, social 
emotional areas; the fourth through ninth categories represent 
neutral, task areas; and the tenth through twelfth categories are 
designated as negative, social-emotional areas. 

The system is general in nature and applicable to small-group 
interaction. Although the system covers the gamut of intaractive 
events that could occur in an interview, albeit in a very broad 
sense, the primary categories of interest in this study are those 
noted as Task Area: Neutral in Figure 1, the information-giving 
and question- asking categories. 

In a transcribed interview user information-giving activity 
is the sum of user units coded in categories four, five, and six 
(gives suggestion, gives opinion, gives orientation). Searcher 
question- asking activity is the sum of searcher units, from a 
transcribed interview, T»rtxich are coded in categories seven, eight, 
and nine (asks for orientation, asks for opinion, asks for 
suggestions) . 
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XPA appeared appropriate for this investigation because (a) it 
is an established «thod of content analysis (Borgatta and Crowther, 
1965; Heyns & Lippit. 1954> Holsti, 1969) (b) definitions for scoring 
the categories 

goried four through nine were directXy related to areas being invest!^ 
gated; (d) "norms" in the sense of profiXes for other 3inalX--group 
Interactions existed (BaXes & Hare^ X965); and <e) the system is 
applicable for future comparative studies of search Interviews ♦ 

Data ColXectlon 

In this section preXimi^nary procedxxres, data coXXectlon period, 
site procedures, collection of evaluation forma ^ selection of tapes 
for transcription > and coding of transcribed tapes are described* 

Preliminary Procedures ♦ Prior to initiation of data collection 
a meeting was held to famiH arize XocaX participating searchers with 
the procedtires and tape-recording process ♦ Out^of --state searchers 
were mailed the procedures (Appendix D), foni»^ and blatUc tapes • 
At this same meeting comments were received concerning the user re- 
quest form and the user evaltiation f crm; some revisions were made 
based upon the input received* Searchers were aware that a study of 
the tape-recorded interviews constituted a part of the research plan, 
but they were not aware of the specific methodology employed (i^e., 
the focus on Inf ormation^giving and question-asking) * 

Data Collection Period ^ Site data was coXXected over a period 
of 4^-1/2 months^ ixam mid-Jtme through the end of October 1978. 
Collection periods for each site were not concurrent ^ since final 
approval of the participant consent form occurred within different 
time frames* Two sites collected data for approximately 3-1/2 
months t one site collected data for a lesser periods 

User evaluation forms were accepted throu^ mid-Deceiid>er 1978* 
The sequence of initial contact by rnail^ with two mail follow-ups 
and a telephoned reminder, required almost two months for completion 
after the end of site data collection^ 

Site Data Collection * Users requesting a MEDLINE search at the 
sites were given a consent form which briefly outlined the project 
plan^ If the user agreed to participate and signed the form tape- 
recording began* Interviews were then conducted in the usual manner. 

Several searchers conwented that they were quite aware of the 
recording process ♦ However » since there was no preknowledge of what 
was being examined in the interviews ^ this does not appear to be an 
Influencing factor in the ultimate consideration of the interviews ♦ 
iftien the tapes were later reviewed by the investigator^ no artifi*- 
ficiality in interviewing procedures was readily apparent. 
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Ac the €nd of the interview tne tape recorder waa turned off. 
If online citations were given to the user the number of citations 
was noted. If the total number of offline citations was known at 
the time, the nxnnber was also noted. In moat cases the total number 
of citations waa not imiaediately known, since backfiles were also 
searched. For these cases citation counts were recorded .when the 
results were received. 

The investigator periodically visited the two local sites to 
pick up tapes, consent forms, and request forms. The non-local site 
mailed materials on a weekly basis* 

During the data collection period a total of 132 interviews were 
recorded at the three sites. Four of these interviews did not result 
in a MEDLINE search because after discussion with the searcher the 
user was directed to an alternative information source; two inter- 
views were recorded but the request forms were not available. 

Of the remaining 126 unique users, 65 were determined to be 
inappropriate to this study. Almost one-third of this number was 
excluded because the search results were not intended for the person 
present at the interview; thirty-five users were excluded because 
they indicated a status outside the scope of this study (e.g., 
attorneys, librarians. Intern-residents). Four of the remaining 
users were excluded even though they were faculty because they indi- 
cated a non-research purpose in requesting the search; five were 
omitted because more than one person was interacting on the tape. 
The residual pool of appropriate users included 23 faculty members 
and 38 students (N - 61). 

Although the recording procedures led to recording of more 
interviews than would ultimately be utilized, this seemed appropri- 
ate. Interaction which would eliminate users at the outset (e.g., 
determination of status, purpose, etc.) is considered in Taylor*s 
five-filter model (1968) to be an integral part of the interview 
process* . 

User Evaluations . All appropriate users were sent a cover 
letter (Appendix E) explaining the project and an evaluation form. 
A return envelope was enclosed. If the evaluation form was not 
returned after 2-1/2 weeks, a hand-written note was sent to the 
user prompting her/him to return the form. If, after another two 
weeks, the form was still missing a third note was sent. As a final 
step an attempt was made to contact the user by phone and urge the 
return of the form. 

A total of 55 forms were returned (90%) ; however, one of the 
forms was returned several months after the cut-off date and was 
not included in the analysis. For the 54 included searches 22 were 
from faculty (96% return) and 32 were student searches (84% return) . 
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Transcription of Tapea . Although it vras origlnp3ly intended 
that Interviews for transcription would be randomly seUcted from 
the pool of cases with coiopleted evaluation fonoB a^d good record- 
ings, this did not prove feasible. Preliminary work with the tapes 
indicated that they would be more difficult to transcribe than had 
been anticipated from an earlier report of similar trans crip tiou 
(Carmon, 1975). A more recant study involving taping of interviews 
in public libraries (Lynch, 1978) suggests that transcription is a 
lengthy » complex process. Xn the present study some familiarity 
with MEDLIl^ procedures and medical temtnolo<<y proved to be neces- 
sary for an accurate transcription of the tapes. For this reason 
the investigator transcribed all tapes rather than delegating them 
to clerical personnel as originally intended. 

Even though the investigator was more familiar with the con- 
text and terminology involved, transcription proved to be a time- 
consuming task, involving in several instances up to 20 hours par 
interview* This was particularly true for those interviews which 
involved use of the terminal while the user was present. Dialogue 
had to be detected against the background noise of the operating 
terminal. Knowledge of the length of time likely to be Involved 
in transcribing each interview indicated that a delay tor random 
c election subsequent to the return of all evaluation forms was not 
appropriate within the time- frame of this study. 

Interviews were selected for transcription by two criteria: 
first, to provide diversity in the searchers and user types in- 
cluded, and second, to allow transcription to occur along an 
extended period. Interviews thus selected were those associated 
with the first faculty and first student evaluation forms retiirned 
for each searcher. This selection aoes not necessarily reflect 
the initial interviews recorded by searchers. Seme searchers iiad 
recorded several inappropriate users before encounterimc a request 
which met study specifications. Other searchers recorded an appro- 
priate user early In the process, and were Involved with an alter- 
nate appropriate user much later on. Two searchers had only student 
interviews available for transcription. A total of 18 interviews 
were transcribed (10 students, 8 faculty). One searcher had con- 
ducted relatively few interviews, none were appropriate for 
transcription. 

The interviews were divided into interaction units according tc 
Bale's methodology (1950, p. 37) at the time of transcription. 

Coding of Interviews . Transcribed interviews were coded by 
two experienced searchers at the Computer Search Service, MIT 
Libraries. Coders were familiar with MEDLINE and other online 
data bases, and as full-time searchers were Involved with inter- 
viewing on a daily basis. It was believed that this famili:irity 




32 



<V,9 



with the Interview process would provide a good background for inter- 
preting the unitized transcripts tn accordance with the scoring 
ays tern • 

The coders were first sent copies of the definitions of cate^ 
gorles (Bales V 1950, pp* 177^195) and copies of several transcripts 
obtained from the Univeisity of Georgia study (Camion^ 1975), so that 
they might become familiar with the coding scheme* Once the coders 
had some practice at applying the scheme to interviews an extended 
training session was scheduled* 

At the end of the training session coders were able to demon- 
strate 82% agreement when scoring three pages containing a total of 
96 units (categories 4> 5, and 6 were collapesed, as were categories 
7, 8, and 9 since data manipulation would involve total information- 
giving by the user) * 

^transcribed interviews for this study were divided between the 
coders so that each received an approximtely equal number of pages 
to Sv.ore* A. duplicate copy of one search was sent to each coder so 
"hat a reliability coefficient for coding material involved in this 
stxidy could be established. Three pages were selected at random 
from the search for the determination* A more stringent method 
(Cohen > I960) than simple percent agreement was applied (see Appendix 
F) . A coefficient of ^Jl was obtained; this appeared adequate for 
the applications involved in this study which focused on the ranking 
of results. 

User infotmation-^giving and searcher question-masking scores for 
the analyzed searches were obtained by counting the frequency of 
events coded by category for the searcher and the user. Topics of 
the 18 transcribed and coded searches are listed in Appendix 
Appendix R Includes an example of a coded interview • 

Data Treatment 

Distribution free tests were used in analyzing the primary 
questions of this study* Specifically, Interrelationships of 
assessment parai!»ters (value with relevance^ satisfaction with the 
proportion relev^t„ concarn for recall, and user perception of the 
searcher in the interview) were examined by means of the Kendall 
rank correlation coefficient* The relationship of user information^ 
giving to relevance and concern for recall scores was also examined 
in this manner • The examination of searcher question-asking as a 
contrib^itory f actorin user assassments of relevance and concern for 
recall was exa^ilned by means of the Kendall partial rank correlation 
coefficient* 
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User differences (faculty » students) were examined by means 
of the Mann-Whitney U test. Earlier discussion suggested more 
stringent assessments by faculty in the areas of relevance, concern 
for recall and perception of the searcher in the interview* and 
greater information-giving activity by faculty in the search inter- 
view; a directional (one-tailed) test was used for these consider- 
ations. A non-directional test (two-tailed) was employed to examine 
differences in satisfaction with the proportion of rel«ivant cita- 
tions and value. 

Version? of the SPSS program was utilized for computations, 
(Nie & Hull, 1977), 
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IV • Results 



Data resulting from this study are considered and analyzed in 
four sections. The first section describes the data sources. The 
second section reports overall characteristics of recorded users* 
The third section addresses the primary questions upon which the 
study focused^ and interprets the results in light of other findings 
related to the scope of this study. The lart section examines the 
interview profiles developed in the coding process > and discusses 
the profiles in relation to this study and other "norms'* reported 
for interaction profiling* 

Data Sources 

General variables considered are available from 126 user re- 
quest forms ^ and the 54 evaluation forms from the 61 users selected 
as appropriate for follow-up evaluations . Respondents included 22 
faculty users and 32 student users. One student did not indicate 
the number of relevant citations r«»trieved»thus the number of cases 
for any testing which includes relevance is 53 • 

Interviews from 18 of the 54 responding users were transcribed 
and analyzed ♦ This subset includes ten student users interacting with 
ten different searchers, and eight faculty users interacting with 
eight different searchers. Factors relating to the analyzed inter- 
views were compared with non-^analyzed cases by means of the Mann- 
Whitney U test. Analysis (P « .05) indicated that there was no 
difference in the distribution patterns for the two groups on the 
aumber of relevant citations retrieved, the relevance scores, the 
satisfaction with the proportion of relevant citations, the concern 
for recall scores^ value scores ^ and the perception of the searcher 
in the interview* Thus it appears possible to assume that the 
searches selected for coding were valid representations of the entire 
sample studied » insofar as these measures are concerned. 

Overall Characteristics 

Although online services have been available for several years > 
more than half (54%) of the recorded users were novice participants 
In an online interview situation. Slightly more than a third (34%) 
of the users indicated par licipation in one to five such interviews; 
10 .3% of the users indici^r^ed they had t^en involved in six or more 
interviews (two users did not respond to this question). The per- 
centage of novice users in this study is greater than that noted by 
Tagliacozzo (47%) in the 1973 study of early MEDLINE users (1975, 
p. 295)^ but less than the 70% new users examined by Carmon (1975) 
in the Georgia/UCIA study* 

As noted previously a significant number of interviews (17.5%) 
occurred in situations where the results of the search were not 
primarily intended for the individual interacting with the searcher* 
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This aspect (second-hand Infonnatlon at the interviev) has not been 
directly examined in this study, but if the proportion of such 
interactions are similarly high in other online operations, an ex- 
amination of user satisfaction in relation to this factor might be 
of interest* 

Since it was believed that some users might consider more than 
one status classification on the request form to be representative 
of their situation, users were asked to designate the one status 
relating most to their request for a search « Approximately one 
third (34«9Z) of the users designated student status (undergraduate » 
graduate, medical.^ dental or law)* Twenty-eight users (22*2%) indi- 
cated faculty ranks (instructor, assistant professor, associate 
professor, professor); almost half of this group were assistant 
professors « Other medical personnel (residents, physicians, nurses » 
dieticians^ therapists, pharmacists) accotinted for 18 « 3% of the 
MEDLINE users* Three individuals indicated post doctoral appoint- 
ments and ten users designated their status as academic researdier» 
This latter category however, was often used by individuals who had 
also indicated that the search was not primarily for their own use, 
suggesting that this category is likely to be chosen by Individuals 
in supporting functions (e*g*^ lab assistants) • Thirteen users were 
less directly involved with the medical field (attorneys or legal 
assistants, librarians, professional researcher, administrator, 
biomedical engineer) . In five cases the status of the user was xan- 
clear* All of these cases involved searches for a different end 
user than the person being interviewed* 

Whatever their status, many of the HEDLINE users indicated that 
their primary reason for requesting a search was research • Almost 
64% of the users indicated a grant project, preparation of an article 
for publication, or ongoing research leading to publication as the 
purpose of their MEDLINE request. If dissertations are included 
72% of the users can be considered to have indicated research inten-- 
tions^ Other academic related reasons included material for instruc- 
tion, masters thesis, term papers or class project (14*2%, Immedi- 
ate clinical applications, where this is defined as use for the 
diagnosis or treatment of a patient or client, accounted for only 
2*4% of the searches. The remaining purposes included seminar or 
talks, legal preparation, curiosity, and compilation of a bibliography 
with no ImmBdiate application* 

In an earlier study of MEDLINE tisers from several institutions 
(Taglacoz20, 1975, p* 296) 38% of the users clustered in what was 
designated as a lower degree category (H*S*, B»A*, B.S*, M.A. ^ M.S.)* 
In this study 54% of the users indicated these degree levels* Twenty- 
three percent of the users are included in a second tier of educational 
attainment (medical, dental or law degree^ two Master's degrees or 
M*D* plus Masters), Eighteen percent of the users had Ph,D* degrees* 
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For 5% of the cases degree status was either unclear or missing 



Although many of the MEDLINE users (68Z) indicated that they 
were already aware of some recent publications related to the topic 
of their search request, only 20% of the users listed one or more 
specific citations on that part of the request form ^ich asked for 
bibliographic information concerning known relevant citations^ 
Fifty-three percent of thxi u^ers said that they were regular readers 
of journals which had articles related to the topic of their request* 
A relatively small niimber of users (13.5%) had already published an 
article on the topic of their search request* Most of the users 
(87%) indicated a continuing interest in the area of investigation • 

In written descriptions of their search needs (the narrative 
statement portion of the search request form) many MEDLINE users 
were not particularly cotimunicative • For the 120 users completing 
this section^ statements ranged from 2 to 76 words (X • 22.3, SD » 16^3)* 
Carmon (1975 ^ p. 33) noted an average of 28*6 words in request state-^ 
ments. When cotanonly occurring verbs > prepositions ^ conjunctions, 
etc. were eliminated in that study, an average of 7*2 information 
conveying words remained* Whatever the reasons, it seems that users 
are unlikely to reveal all of the parameters and specifications of 
their search request by means of the written search statements If 
this is true, clarification of the request in the search interview 
would appear to be of considerable importance. 

Search Results 

Earlier these questions were posed: 

1* How do specific user evaluations in the areas of 
relevance, satisfaction with the proportion of 
relevant citations retrieved, concern for recall, 
and perception of the searcher in the interview 
related to the user's indication of the value of 
the search results in meeting the need pron^ting 
the search request? 

2* Do students and faculty MEDLINE users differ in 
evaluative designations, and in information-* 
giving during the interview? 

3* What are the interrelationships between user 
inf ormation^-giving , searcher question- asking 
and two traditional foci of assessment ^- 
relevance and recall? 

This seition considers the results associated with these questions* 
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l uterrelatlonshlps of User Assessments 

When the evaluation scores are ranked and compared to the 
ranked value scores by means of the Kendall rank correlation co- 
efficient* significant correlations are noted in each instance. 
Table 1 indicates, in decreasing order of relationship, the tau 
values observed for the cases. 



TABLE 1 

Correlations of Evaluative Measures 
With User Designations of Value 



Measure Kendall »s tau 



Satisfaction with the proportion 
of relevant citations retriev 

Concern for recall .$0 



p " .001 



of relevant citations retrieved ,66 

* 



Relevance ^47 



Search interview (User perception of Searcher) .42 



* 



Like users in most other reports of online evaluations, 
MEDLINE users, when considering the value of the search results in 
meeting the need prompting their request, indicated generally 
favorable responses concerning value. This was true even though 
users were provided a ten-point scale to indicate value rather than 
the more customary three- or four-point scale. No users marked the 
lowest designation (no value) and almost 60% of the users indicated 
a value of eight or higher. Figure 2 outlines the distribution of 
responses on the value scale. Application of an extended scale 
does, however, appear to have some merit in increasing the variation 
of response patterns, which is particularly useful if comparisons of 
magnitude, as, in this study, are desired. 

The favorable response pattern on value is coupled with a 
fairly low pattern of relevance scores. The mean relevance score 
for all searches is 45%. Thirty-one of the searches had relevance 



38 



N - 54 



15% 



20% 



(0 



§ 

u 
u 



15% 



10% I 



5% 



-f r 



ii 

1 1 



1 1 



8 



10 



Value 



Figure 2. Value of MEDLINE Search Results in Meeting the 
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scores of 50Z or less (i.e., in Idiese searches half or fewer of the 
tltT""^ ?° ^^^^ considered to be relevant to the 

''^'^^^ • ^^^"^^ ^ """^^^ distribution of reLva^ce 
scores. The distribution tends to be U-shaped, a pattern^iT^s 
noted by Rees and Schultz (1967, Vol. 1, p. 118) «hen th^y po^e^ 

rei::rc:."''"'' ^^^^^ 4.ri::nt^'^?xif 

mod^TtVl^t^r.^^'iT^^* relevance and value scores sho*. only a 
S su^esS^^^^ (r--.47). Tagliacozzo (1977) 

nas suggested that dissonance in user responses may occur- in i-har 

us'^f ^:inLl^^^^^^ ^^^^r^^> noted"^L1e Wness^nd 

usetulness. In the present study dissonant responses appear to be 
concentrated in cases with lower relevanca scores For ex^Je 
all cases of medium (5 to 7) to high (8 to 10) value dlsWioAs 
are associated with medi^^ to high relevance sco^s but SwS 
relevance scores (0^33Z) are associated with low to medi^T and 
high value designations. Relevance scores are simply rat?^s deter- 
of'c^taLo^'r^^^^^ "'^"^^^ citations'b^ the totfl nL>er 

cLat^ons^. f T"^ "^^^'^^ ^'^'^^ ^^^^ of "good" 

Alt tZ ? ^"""^ ^^^^'^^^ example, a user might 

find four relevant citations out of a total of five retrieved- the 

"^ TZlllZl r^^^ ^^^^ ^^^^ '^-^ reSv:nf ^ita^ions 

^.t results in a total of 80 citations; the relevance 

r:!::an^ '%Ti'' — ^ ^-^ver, has ^l^^r 

lltl Zi -citations. It has been suggested that searches that produce 
^977 J IS? references are associated with helpfulness (Tagli™, 
1977 p. 246) or satisfaction (Kobelski & Trumbore, 1978 o 16^ To 

^s^liTt^''^^^ f ^ --^-^ reierenc;s';ere^;on-' 

ness scal^ ^n'^h/f ^"^^ '"^"^ ^ dichotomized helpful- 

ness scale. In the second case the authors noted that all users re- 

T.^ '^"^'"^^ ^'^^^^^^ ^^^^ satisfied Sey^So 
i^re satisfid "'reiving total outputs of 100 or more citations 

citat^ons'''^ ?I^rf ^^.'^^^--^'^ «<^eived 40 or fewer relevant 

cications. Figure 4 indicates the distribution of relevant citations 

ano^arL'b'"^ f citations, the results in the currSt ^"iy 

Zry^k ia^?f ^'t ^i"" "^"^^ indicated in the public library experL 

Trl if f 1 ^\ ^ ^^^^ ^'^''^y ^e^^^^'i year «sers recSved 

0 to 15 relevant citations, in the third year 75% of the users received 
0 to 15 relevant citations. Yet it can be recalled that 68% to 76% of 
vaS: '^-^ ^^eir 3-rch was of major or coiid^rlble 

7t the actual number of relevant citations 

ZtltZl I ""^t '^'^ ""^^ paramount importance in value esti- 

^ittZ: '"''"'^''^ "^^^^ ^en the number of 

relevant citations received are ranked and compared to ranked user 
value designations a moderate relationship (T". .53) similar to that 
noted for relevance is observed. A negligible relationship (T** 16) 
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Figure 3. Distribution of Relevance Scores for 
MEDLINE Searches. 
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in the MHJ* INE Searches. 



42 



ERIC 



i9 



appears to exist between the total niiaber of citations retrieved 
and value. 

The obtained search interview score (the mean score confuted 
for five questions to the user about the searcher's role in the 
interview) showed the least relationship to the value score »42). 
User impressions of the searcher in the interview situation were 
generally favorable. Scores of 7 or higher were observed for 77% 
of the searches. Figure 5 indicates search interview scores for the 
lffiDLI5lE searchers. Findings reported here do not appear to be in- 
consistent with those reported earlier by Gothberg (1975) concerning 
user perceptions about two types of interview situations and user 
satisfaction with the transfer of information. Users exposed to 
immediate interviews (a sense of "liking" is conveyed) were more 
satisfied with the interview itself and better satisfied with their 
own performance in the interview than were users exposed to non- 
isroediate interviews. However, the two user groups eadiibited no 
difference In their expressed satisfaction with the transfer of 
information. Although the conditions are different, one might 
anticipate from the earlier study that the user perception of the 
interview (a social interaction) will exhibit a somewhat lesser 
relationship with a judgment of value than other areas more closely 
aligned with the product (the search results) being evaluated. 

Most closely associated with the value designations indicated 
by the user were the scores on satisfaction with the proportion of 
relevant citations retrieved ( 7"* .66) and the mean score, concern 
for recall (T"» .60). Some of the dissonance noted at the bottom 
third of the relevance scale (high, medium and low value scores) may 
be related to a perception by some users that 1- ts^r relevance means 
that they have missed some items. Eight of the nine users indicating 
a satisfaction with the proportion relevant score of three or less had 
relevance scores lower than 30%. Five of these eight had concern for 
recall scores less than three. 

In the Georgia/UCLA study (Carmon, 1975), users ranked the lack 
of ability to judge completeness as the most disliked feature of the 
search services. In the current study the concern for recall score 
exhibits a considerable relationship with the value score. Repetition 
in this study of a situanion in which users appear to attach some 
importance to the recall aspect of a search is particularly interesting 
in light of Cooper's suggestion (1973) that recall may be an inappro- 
priate measure for consiaeration in retrieval evaluations, i.e., he 
does not consider it Important to the user because, in a general sense, 
the user cannot "know" that he has missed citations Tn the results.. 
Figure 6 indicates concern for recall scores. Figure 7 indicates the 
satisfaction with the proportion of relevant citations. 
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Figure 5. ^stribution of Mean Scores for User 
Perceptions of the Search Interview* 
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Figure 6. Distribution of Concern for Recall Scores, 
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SATISFACTION WITH THE PROPORTION OF RELEVANT CITATIONS 



Figure 7. Distribution of Satisfaction with the 
Proportion Relevant Scores, 
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Us»ar Differences 



Results were examined for differences by user type (faculty, 
students) ir4 the areas of relevance, concern for recall, perception 
of the searcher In the Interview, and information-giving activity 
(one-tailed tests) . Differences in value and satisfaction with the 
proportion of relevant citations were also considered (two-tailed 
tests). The Mann-Whitney U procedure (Siegel, 1956 » p. 116) was 
used in the analysis. Table 2 provides summary results. 



TABLE 2 

SuniLary Results for Testing Differences 
by Type of User (Faculty, Student) 



Mean Rank by Group 
faculty Student 



Measure 


(il = 22) 


(N = 32) 


U 


P 


Relevance 


28.00 


26. 


319.0 


.35 


Concern for Recall 


32.39 


24.14 


244.5 


.03 


Perception of Searcher 


27.64 


27.41 


349.0 


.46 


User Inforcvation- 
Giving 


8.31 


10.45 


30.5 


.20 


Value 


30.52 


25.42 


285.5 


.24 


Satisfaction with 
Proportion of 
Relevant Citations 


31.52 


24.73 


263.5 


.12 



31 students 

analyzed interviw — 8 faculty, 10 students 



Earlier reports (Rees and Schultz, i%7: Saracevic, 1970) pro-- 
vlded a basis for belief that, in relevance assessments * knowledge-* 
abilty tnay be associated with more stringent assessment* In the 




current study these findings were extrapolated to a model which 
suggested that this same stringency of assessment would occur in 
actual searches of an online data base* Specifically, it was 
suggested that a more knowledgeable group of biomedical users 
(faculty) would indicate lower relevance scores, lower concern 
for recall scores (be more concerned about recall) , and lower 
searcher perception scores than would a less knowledgeable group 
of users (students). Since value and satisfaction with the pro- 
portion of relevant citations retrieved had no Imiaediately apparent 
connection with knowledgeability , it was suggested that the two 
groups would exhibit no differences in these areas. When the 
interview itself was considered it was thought that the more knowl- 
edgeable usftts would exhibit greater information-giving activity. 

As anticipated, no difference in user groups was noted on the 
two jDore general measures of value and satisfaction with the pro- 
portion of relevant citations. However, in areas where it was 
expected that faculty users would make more stringent assessments « 
this did not occur. Faculty and student users exhibited no signi- 
ficant rank order differences in the direction predicted for rele- 
vance scores, recall scores, or search interview scores. It is 
interesting to note that for users in this study, students actually 
appeared to be more concerned with recall than did faculty users 
(P * .03). In information-giving activity faculty users exhibited 
ao significant rank order differences when compared with students. 

Since the expectation of more stringent assessments and more 
information-giving activity from faculty was based on the supposi- 
tion that faculty users were more knowledgeable, several items from 
the user request form were examined to determine if indicators of 
knowledgeability were present in the sample studied. 

Utilizing responses from question 12 on the request form, 
c'sgrees completed by the faculty and student users were examined. 
To Che extent that knowledgeability is equated with higher levels 
of educational attainment, faculty users can be considered more 
knowledgeable when the proportion of users in each category is 
considered (Kendall's tau C - ,94, p - < .001). Ei^teen students 
had completed one degree, the Bachelor's; eleven students had an 
undergraduate degree plus the Master's, these groupings were des- 
criptive for none of the faculty. Ten users had completed a pro- 
fessional degree program (M.D., J.D., D.D.S.) or had two Master's 
degrees. Three students were in this category. Fifteen users had 
a Ph.D. or a Ph.D. plus another degree (e.g., M.D.). No students 
were included in this category. 

Responses to question 13 on the request form were examined for 
variables related to general knowledge of, and continuing interest 
in, the topic of the search. No significant associaton is noted 
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between user status and the awareness of several recent publications 
on the topic of the request (19 out of 21 faculty responded affinna^ 
tively^ as did 23 cut of 32 students) » or between status and the 
user^s continuing interest in the topic (all faculty indicated con^ 
tinning interest, 30 out of 31 students indicated the same interest)* 
However, faculty users were more likely to read on a regular basis 
journals on the topic of the request, and to have published in the 
area of the request. Table 3 indicates journal reading habits > and 
Table 4 indicates responses concerning publication* 



Table 3 

User Type by Regular Reading 
of Journals 





Read 


Journals 






Yes 


No 




SCudent 

• 


13 


19 


32 


Faculcy 




16 


6 






29 


25 


54 



X" - 4.19 ; p < 0.05 



Table 4 

User Type by Publication 
of Related Article (s) 





Have 


Published 






Yes 


No 




Student 


3 


29 


32 


Faculty 


8 


14 


22 




11 


43 


54 



X*^ - 4.31 : p 0.04 
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^ topic the evalxiated searches could be subjectively divided 
into two areas, conventional biomedical searches, and those more 
directed to the social sciences (e.g», spirituality of patients, 
sexuality and the mentally retarded) , No systematic relationship 
between type of user and area of search was apparent, and when 
evaluative measures were contrasted (relevance, satisfaction with 
the proportion of relevant citations, concern for recall, value, 
and perception of the searcher) there was no significant difference 
(p » i .05) in the distribution of evaluative scores by topic area 
of the search. Table 5 indicates stmmary results for testing evalua- 
tive differences by area of search. 



Table 5 

Sumnary Results for Testing Differences 

by Subject Area of Search 
(Biomedical . Social Science) 

Mean Rank by Area 

Biomedicine Social Science U P 
Measure ^ (N » 41) (N » 13) (two- tailed) 



Relevance 


27.67 


24.71^ 


218.5 


.56 


Satisfaction with 
proportion relevant 


29.61 


20.85 


180.0 


,08 


Concern for recall 


29.57 


20.96 


181,5 


.09 


Value 


28.95 


22.92 


207.0 


.22 


Perception of 
Searcher 


27.71 


26.85 


258.0 


.86 



a =• 12 cases 
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Faculty users were more likely to have participated in a 
previous online interview than were student users. Seventeen 
faculty users (77.2%) had participated in one or more such 
interviews; twelve students (37.5%) had previous experience with 
online interviews. However, if cases are contrasted by previous 
experience, or no experience there is no significant difference 
in Che distribution of evaluative scores for the two situations. 
Table 6 indicates summary results for examining distribution 
differences by experience with online interviews. 



Table 6 

Summary Results for Differences 

by Previous Online Interview Experience 
(None, One or More) 



Mean Rank 



Measure 



None 
(N » 25) 



(One or More 



U 



( two-tailed) 



Relevance 


25.10 


28.70^ 


302 


.5 


.40 


Satisfaction with 












proportion relevant 


26.68 


23.21 


342 


.0 


.72 


Concern for recall 


27.64 


27.38 


359 


.0 


.95 


Value 


27.08 


27 * 86 


352 


.0 


.85 


Perception of 












Searcher 


29.23 


25.97 


313 


.0 


.44 



a 28 casas 



Previously it was mentioned that some interview sessions in- 
corporated the terminal search as part of the interview process. 
There was no apparent systematic relationship between user type 
(faculty, student) and participation in a particular type of inter- 
view, yhen interviews including the terminal session are contrasted 
with non-terminal interviews an interesting difference in the area 
of the perception of the searcher in the interviews is noted. This 
suggests that, for this study, users in terminal session interviews 
assigned higher values in the area of perception^ of the searcher. 
Table 7 outlines results of examining terminal and non-tenninal 
interviews on evaluative measures. 



TABLE 7 

Summary Results for Testing Differences 
by Type of Interview Session 
(Terminal, Non-Terminal) 





Mean 


Rank 






Measure 


Terminal 
(N - 15) 


Non-Terminal 
(N » 39) 


U 


P 

(two-tailed) 


Relevance 


33.00^ 


24.35 


189.0 


.09 


Satisfaction with 
proportion relevant 


32.80 


25.46 


213.0 


.12 


Concern for recall 


34.07 


24,97 


194.0 


.06 


Value 


29.37 


26.59 


257.0 


.49 


Perception of Searcher 
in Interview 


37^7 


23,59 


140.0 


.003 



a » 1-+ cases 
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Information Giviag> Question-Asking 



This section examines the interrelationships of the user in^ 
tonaat ion- giving during the search interview and searcher question- 
asking activity with two traditional areas of evaluation concern 
(relevance and recall) . Recall in this study is derived concept 
"concern for recall" rather than an actual ntamerical determination. 

User information-giving activity scores (the sum of user acti- 
vity coded for categories 4^ 5^ and 6 of the Bales scheme) from the 
IS analyzed interviews were ranked and compared to user designations 
of relevance and concern for recall by means of the Kendall rank cor- 
relation coefficient ♦ Information-giving activity was si^ificantly 
related to relevance; however, the relationship to concern for recall 
was negligible and non-significant* Searcher quest ion- as king activity 
scores (the sum of searcher activity coded for categories 7, 8» and 9 
of the Bales scheme) were significantly related to user information- 
giving scores • Table 8 outlines the results from this examination. 



TAB1*H S 



Correlation of User Information-Giving with 
Relevance^ Concern for Recall* and 
Searcher Question-Asking 



Measure 



Kendall *s tau 



Relevance 



*50 



Concern for Recall 



.20 



Searcher Question-Asking 



o8 



p « .001 
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Earlier it was suggested that information-giving by the user can 
be either voluntary or searcher elicited. To examine the remaining 
relationship between information-giving sad relevance when the effects 
of question-asking are held constant, the Kendall partial rank corre- 
lation coefficient was determined (Siegel, 1956, p. 223). The partial 
rank correlation coefficient (relationship of information-giving with 
relevance with the effect of question-asking held constant) was .38. 
The decrease in the relationship initially observesJ (.50) provides 
some basis for suggesting that question-asking by the searcher moder- 
ately enhances the relationship between user information-giving scores 
and relevance scores. 

The volume of information-giving units and question-asking units 
varied considerably over the evaluated interviews. The mean score for 
searcher question-asking units was 37,5 (SD 27.3); the mean score for 
information-giving units was 80. S (SD 59.6). 

Seven of the eighteen coded interviews included terminal sessions. 
More question-asking was noted for_these interviews (X « 62. 7» SD 26.3) 
Chan for non-tarminal interviews (S » 21.5, SD 11.1). More information- 
giving was noted for users in terminal searches (X « 119.6, SD 51.6) 
than in non- terminal searches (X » 56, SD 52.1). 

.\lthough relevance itself has a moderate relationship with the 
value of the search results ( f"- .47) and information-giving appears 
to be more directly associated with relevance (f* .50) than does 
question-asking ("T^* .36), some interactive effect appears to occur, 
which suggests that training in question-asking may be a factor for 
consideration in preparation of online searchers. To the extent that 
question-asking is important, the impetus, or opportunity for additional 
questions appears more likely to arise in interactive searches at the 
terminal with the user present, than in interviews removed from the 
actual terminal search session. 

Interview Profiles 

In this section an overall profile for all coded searches is 
presented and compared with a standard suamary profile which rep- 
resents pooled findings from 21 studies which employed the Bales ' 
scheme (Bales & Hare, 1965). Breakouts for searcher and user inter- 
action, and interaction by type of interview (non- terminal, terminal) 
are also discussed. 

Figure 8 gives the percentage of interaction by category for the 
MEDLIiffi interviews. Darker vertical grids indicate points ' SD above 
or below chose noted by Bales and Hare in the pooled interaction 
studies . 
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The taost obvious departure occurs In the Inf ormatioti-giving 
category. Agreement units and asking for suggestion units are also 
higher (h- X SD) than those reported previously. This pattern would 
appear to be consistent with the aim of the search interview *'to 
arrive at a clear ^ narrative > natural language statement of the 
user^s information needs and to gather a nunijer of facts and clues 
to be used to amplify or refine this statement" (Carmon, 1975, p*4). 
The profile is generally lower in those areas which indicate positive 
and negative emotional areas* Some of this may be attributed to the 
method of coding, i.e^, utilisation of a typed-transcript • Waxier 
and Hishler (1966) noted lesser use of negative categories in typed 
transcripts (loss of emotional tone in interchanges) but found that 
an overall comparison (rank order) of category use for the two methods 
(typed transcript^ tape and typed- transcript) g ve similar distribu-- 
tions* The lower rates noted for categories 1 and 2 may simply reflect 
the business-like nature of the interview situation* The higher percent 
of units scored for the agreeioent category (Category 3) parallels find- 
ings in other types of discussion groups where participants are required 
to reach consensus on an issue • 

Figure provides more illxjmination on who does what during the 
inter^/iew* It is apparent tha^ the searcher dominates the task area 
of the profile (Categories 4 thiough 9). The information-giving 
activity by the searcher is most striking* since it seems that this 
giving of information is not necessarily the response to a high rate 
of quests --asking activity by users. Users appear to take a rela- 
tively pa;^aive role in asking questions* If one of the aims of the 
interview is to reach a consensus on the needs of the user^ it seems 
that the consensus that is reached may be somewhat influenced by the 
:iearcher* The searchers indicate higher levels of giving suggestion 
and opinion^ while users are predominant in the agreement categoiry* 

Earlier it was noted that more information-giving and question-- 
asking was present in terminal searches* Profiles of percent of 
searcher interaction by category, for total searcher interaction by 
interview type (Figure 10) show fairly similar proportional patterns 
for searcher interaction* Searchers in non-^terminal interviews ex- 
hibited more proportional activity in the agreement category* Sear-- 
chers in terminal interviews exhibited more proportional activity in 
asking for orientation ^ or facts from the user^ but somewhat less in 
asking for opinions or suggestions. Terminal interviews appear to 
create more tension or withdrawal situations for searchers than do 
non-terminal searches. 

Users t like searchers^ also have similar interaction profiles in 
the two c:/pes of interview (Figure 11) . Users in tertninal interviews 
exhibit a somewhat higher proportional activity in the positive (Cati^- 
Tories 1 through 3) and negative categories (10 through 12) than do 
users in non-terminal searches* 
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Although reference negotiation has been considered as a process 
in which the intermediary *' interrogates'* the user (Taylor^ 1968) > the 
profiles outlined here apoear to suggest that information-giving is 
the prime activity for both searchers and users participating in an 
online interview* In a quantitative sense more information-^giving 
by the user and more question-asking by the searcher appear to occur 
in interviews which are combined with the actual terminal search. 
Whether the increased activity is a function of specific events asso-* 
ciated with terminal sessions or only the increased amount of time the 
interviewer and searcher spend together is not readily apparent. 
However » terminal sessions may provide the atmosphere for more helpful 
information exchanges simply because they imply an extended interview 
session. From a subjective perspective it seemed that users in terminal 
sessions volunteered several important background characteristics about 
themselves either while waiting for the terminal to respond to some 
particular input* or as a result of particular citations they viewed 
in che session. Searchers appeared more likely to question users 
about seemingly dissonant responses to a particular citation > e.g»* 
asking why a citation was not relevant when it appeared consistent 
with something the user had previously asked for> or alternatively 
asking the user why he like a particular ciation when it did not 
seer^ to be within the scope of material previously discussed. 
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Con Illusions 



Conclusions resulting from this descriptive study emanate from 
sevtiral sources, from data tests; from trends evident in supplemen-* 
tary analysis of xhe data, and from subjective impressions developed 
by the investigator in the process of the study* The primary limiting 
factor affecting generalizability to other similar academic MEDLINE 
search sites ^ would appear to be the self ^-selection of users who par- 
ticipated in this study. Such selection is likely to be a continuing 
factor in the future for similar studies as very proper concerns for 
the privacy of individual subjects are expressed through the applica- 
tion of increasingly rigid procedures to protect subject rights. 

For users in this study the satisfaction with the proportion of 
relevant citations retrieved^ and expressions of concern in the area 
of recall for the search, appeared to be more importantly related to 
the value of the search results than did the actual proportion of 
relevant citations (the relevance score) • This suggests that greater 
elucidation of the individual user 's tolerance level for irrelevant 
citations at the outset of the search process may be important. This 
aspect is sometimes covered by a broad-*narrow question on the request 
form (e.g-^ Do you want a broad search retrieving many of the relevant 
citations J but which may also retrieve many irrelevant citations? or^ 
Do you wane a narrow search retrieving primarily relevant citations ^ 
with few irrelevant citations^ but which may exclude some relevant 
citations?) A question of this type would also appear to have some 
bearing on establishing the user*s potential concern for recall* 
This type of question was not included on the request form for this 
study, so no specific examination of pre-search expressions of 
preference in these areas can be made* Specific elucidation of the 
user'si preference during the interview is suggested because, in general, 
for interviews considered in this study, the user request form, if 
referred to at all^ appeared to serve primarily as a source of informa- 
tion concerning the subject of the search. In many cases the form was 
noc completed by the user until the end of the interview* 

In a formal sense ^ results provide no basis for suggesting that 
trie nresumed greater knowledgeability of faculty users (with some 
justification for the presumption of knowledgeability evidenced in 
the higher degree levels attained by faculty ^ the greater likelihood 
faculty to read on a regular basis journals related to the search 
topic, and the greater likelihood that they had published on the 
topic) is a factor leading to more stringent assessments in the areas 
of relevance » concern for recall, and perception of the searcher in 
the interview. Knowledgeability in the sense considered here also 
vioes not appear to be a factor leading to greater information-giving 
activity In the search interview. 

Interaction profiles developed in this study showed that in- 
:\?rTa**t ion -giving activity (as defined by Interaction Process Analysis ) 
instituted tr.e largest proportion of all searcher activity. Tliis 
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observation repeats in a sense the observation by Lynch that *^parti- 
cipating librarians /in reference interviews in public libraries^/ 
spent a significant proportion of their time giving information to 
patrons rather than getting information from them" (1977 > p. 129)* 
Mo direct examinations of searcher information giving and any assessment 
measures were siade in this study* However » it does appear that some 
relationship exists between user informations-giving activity and 
relevance scores* Qvxestion^ asking by the searcher has been suggested 
as a contributory factor in this relationship* 

Users in terminal session interviews exhibited more information- 
giving activity than those in non--terminal interviews* Searchers in 
terminal interview sessions e^diibitea more question-asking activity 
than did searchers in non-terminal sessions* A trend in the data 
results suggests that users participating in terminal sessions assigned 
somewhat higher relevance scores^ satisfaction with the proportion of 
relevant citations scores > concern for recall scores (were less concerned 
about recall) ^ and search interview scores than did users in non-terminal 
sessions* A subjective impression received by the investigator in 
listenxng to the tapes of the interviews is that the user^s viewing of 
actual citations during the interview prompted interactive activity 
(inf oraacion^ giving » agreements, disagreements ^ and question-asking) 
that may have been helpful in the search process* A further examination 
of the type of interview (terminal* non-tera"'.nal) and assessment measures 
nray orovide additional information for improving the interactive and 
iiagncstic process* 
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Appendix A 
CONSENT FORM 



OAKLAND L N t V £ R S J T Y 



{t3ch««t«r, Michigan 43063 



A research study supportea bv the Office of Libraries and Uaming Resources, 
Ottice ot Education is being conducted at the Houston Academy of Medicine - Texas 
Mecical Center Library, the University of Michigan and Wayne State University. 
This Droject is directed to an investigation of two areas: 1) the relationship 
of user characteristics and search ourpose with the user judgement of MEDLINE 
search results and 2) an examination of the MEDLINE search interview for the 
identification of variables related to user judgment of search results. Primary 
data collected for this study include user information supplied on a MEDLINE 
request form;^ tape recorded interviews for MEDLINE searches; typed copies (all 
iaenti:yins? characteristics deleted) of inter'riews from respondents completing 
.1 user evaluation form; and user evaluation forms returned by selected respondents. 
It i-. anticipatea that the results of this study may provide" a basis for clarifi- 
cation of performance objectives for on-line information retrieval systems. As a 
:urrent user, and a potential future user, of such systems your cooperation in this 
project is requested. 

Your participation involves three areas: 

1. Conplecion of a search request form. 

Participation in a tape-recorded interview session. Should you 
agree to recording, you may change your mind at any time during 
:he interview and request that taping cease. 

Potential selection as a respondent for a twelve-item evaluation 
forr.. Items include eleven scale responses and a count of rele- 
vant citations in your print-out. 



3. 



The i>:entitios of all participants in this project will be held confidential 
3v the investigator. 

r irticioation in this project is voluntary, non-participation will not affect 
rrisatnent :f your MEDLINE request. Participants may withdraw from the project 
it -inv time. 

Eileen E. Hitchingham 
Project Investigator 



September 6 » i9"8 



I -.CILLING TO PARTICIPATE TN THE PROJECT OUTLI.NED ABOVE. 



Aini :e:^ ter 
."Mte 



Sea rcher 
Date 



the tUTure, i? rmER -qualified investigators should request copies 
.•:r.v r.\?E3 and or completed request ^orms, i agree to their release by the 



DC. < 



lECT investigator IF ALL XA:-IE .A^JD INSTITUTION REFERENCES ARE DFLETED. 



Aif; tester 
0U^» 



Searcher 



Date 
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Appendix B 
REQUEST FORM 



MEDLINE 3IL\RCH HJEQt^ST FORK 



Mo , Day / Y«ar 



Agr«« to Tap« "'as 

Search Number 

Citatlona On-liae.^ Of f-lin« 



2* t^Xaphon* Number 

3* HaULiag ^iddreas 



/ 



Area Code / Number / Exraasioa 



City / State / 



lip 



6* Huaiber of pravtotis oa-Uae search latar-zi 



aws. 



the results of this search are orlaariiy 
tor ay owti use* 



3. If S0» please Indicate Naae. title and Address 
of person for whom Search la intended: 



^Tone 
1-5 

More than 10 

Yes 



Mase 



Address 



City 



otaca 



::io 
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MESLIJIE SEARCH REQUEST FORM 



-» 



When search is completed: Send resales by mall 

I will pick up results 



Check Che ONE blank which BEST describes che requester or the 
search. If two designations appear to be applicable (e.g. a 
Physician who is also a Graduate student) check the status which 
relates MOST to your request for a search. 

EXAMPLE: 

A Physician/Graduate student initiating a MEDLINE request 
PRIMAHILY for an actual patient care problem would designate 
"Physician." 

A Physician/Graduate student initiating a MIDLINE request 
PRIMARILY tor graduate-related coursework would indicate 
"Graduate Student." 



■L. . 



Uadargraduate student 

2* Medical student 

3* Graduate student (Master's level) 

4, Gradxiate student (Ph.D. level) 

5* Instructor (includes Clinical S Adjunct appointments) 

6, Ass*t* Professor (includes Clinical & Adjunct appointments) 

7. Assoc* Professor (includes Clinical i Adjunct appointments) 
8* Professor (includes Clinical & Adjunct appointments) 

9. Postdoctoral appointment 

10 . Intem/Sesident 

11 • Academic researcher 

12* Physician 

13* Nurse 

14* Dentist 

15. Pharmacist 

16* Other (please specify) 
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MEDLINE SEARCH REQUEST FORM 3 

11. Ch»ick chft ONE blaak which BEST describes your purpose in 
request ia'i a MEDLINE search. 

- Grant project (development* in progress, completion) 

. Preparing an article based on research work for 

publication 

. 3. Preparing a review article for publication 

^- On-going research which will lead to publication 

5. Dissertation (Doctoral degree) 

6, Thesis (Master's degree) 

Tera paper 

3. Class project 

9. Background taacariai for a seminar 

10 » Background oaaterial for a speech or talk 

. 11. Instruction or teaching 

Clinical application (diagnosis, treatment of a 

patient or client) 

Personal b -bliography , no l^edlate application 

. 1**. Other (please state) 



Check ALL degrees completed 

_____ 1. High school 

Associate's degree 

3. Bachelor's degree 

Master's degree 

5. Doctoral degree 

. 6. M»D. 

7. D.D.S. 

. 3. Other (please state) 
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ME:)tlHE SEARCH REQUEST FORM 



13. ?I«4se consider the foUcswiag statemeacs aschey ^pply 
so the copic of your Search rftqua<it. 

1. I am aware of several receac (laac 2 years) 1. ?es 

pubiicacions related to the topic of ay Search No 
request . 

r read on a regular basis (weekly, aotithiy) 2. Tes 

several journals which have articles re- No 
lated to the topic of ay request. 

3, t have published an article or articles* 3. Tes 

reiatea to the topic of ay Search request. No 

The topic of this search request is likely 4. 

to be of coutiaiiing interest to ae (next No 

one or two years) . 



Searcn request: Please give a detailed statement of the 
subject aattar for which the Search is to be conducted. 
Define any terras which aay have special aeaning relative 
to your Search and/or any concepts you wish to exclude. 



MEDLINE SmCK 5IQUESt FORM 



>^^l«^vanc ^isatloQs ^ If possible Use auchor, cicle and 
pubiicacion daca for any known relavanc articles published 
wltaia cr.e lasc chree y«ars. The citations will be used as 
a gtiide for recrievlag siaiiar cicationa related to your aeeds. 

1. Author 

Title 

?-ublication Data 

2. Autnor 

title 

Publication Data 



3. Author 

Publication Data 

For Use in VAYNE Searches Onlv. 



t^CR LIBH^RV USE ONLY. DC NOT miTZ IN THIS BOX 

Library Data 



2; '^Su for:nulited search 

Search unites) 

-ine t;ae 

Fornt ti:se 

: C ?t^-f^™^l^t^<i search 

' Search unxt 

wine charges 

22 Institution ran search 
Minutijs on-ixne 



Recurring search 





Monthly 


Method 


of payment 




Free q 


a 


IR3 Q 


a 


To be invoiced 


Search 


forniuiated by 


Search 


run by 



Cuartsrlv 



Q Check ( S_ 



Date run 
Taskname 
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« w t a ^ w *^ ^ ^ ne ^r xn t s « 
MEDLINE 

BAcr: 

3ACK59 
r"HER ; specify) 



per cit 
per cit 

C <db 



;uXc.«^L^ 
aNCSRLIXE 
EPILEPSY ABSTS." 



r- ^ ' 

per 



6 



Specify riles to be searched : 

Q MEDLINE {latest 2-3 years) 

rj 3ACKr: (1972-74) 

C 3ACX69 (1969-ri) 

- EACKoo (196t)-(58) 

i;;;^ Other (specify) 



Q TOXLINE (l?7l to present) 

Q TOXBACK (1940-70) 

gCANCERLIN'E (1965 to present) 

Q EPILEPSY ABSTS. (1945 to present) 



Search liaitations : Please check all boxes and supply information oertinent 
to tr.e scope of your search. 

1 . <;;3 No restrictions 

^ Human only 



a. — Mai 



b. ;2 *^ifai^t> newborn (to 1 mo.) 
Infant (1-23 njos . ) 
;3 Child, preschool (2-3 yr-i.) 



.-Adolescence (15- IS yrs.) 
n Adult (19-44 yrs.) 
r Middle age r4S-64 yrs.;; 
Q Aged (•JL- yr:^ . and older) 



3. Anisial only. If only certain anisnals or ani.'aal gtr^ups, list below; 



i. 



e. 



4. Language restrictions: 
r; Accept all languages 

English abstracts 



Other languages accepted (specify) 
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Search romulation 



Pertinent MeSH headings or keywords: 



0. 



Search btatements: rOR LIBRARY USE ONLY 



a. 
u. 

14. 
IS. 
16. 

1 *? 

13. 
19. 

:o. 



Off-line 



\ 

I 

m 

& 

1 



4. 



5. 
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Appendix C 
EVALUATION FORM 



Search No« 



MEDLINE USSR RESPONSE FORM 

You have been selected from a number of users who participated 
in a taped interview for a recent MEDLINE search. Your cooperation in 
:on:pIeting tnis forta as part of the follow'-up analysis of user juagment 
2I results is sou^hc* 

The questions consider four areas: 

1* AeK^vance — Your jud^tnent of each cicacion retrieved in 

the MEDLDtE search and its logical connection (relationship) 
to t:ie question ;/ou posed to the system* 

1. Recall — Vour judgment of the "completeness" of the search, 
i,e*, are you satisfied that the search retrieved all of the 
relevant citations in the MEDLINE data base? 

V *i I u e f the 5 e a r c h — How valuable was the search in :?.eeting 
tne inforriation need whicn prompted your request? 

Search I n t e r v 1 e w Your judgrnent concerning the ability of 
the searcher tD assess and explore all aspects Dt vour in- 
f,^r:r.acien need during? the search interview. 

An^awi r ALL :;uestions as best you can and reel free to make iny 
:.^•n^.e^t3 v thiuk nocossar-*. Your answers and cotnnients will be kept 
1 i ien t i a I » 



?l-.v2se complete this forr: and return It within two weeks to: 

Eileen E. Hitchingharrj 
<r^sa:e Lit)rary 
"^akland Universitv 
Rochester Michigan -306 3 

A ^trrp^^i envelooe is enclosed :^r your -ise* 

PLiase rum to questions :n the rollowin!? pa^i^s. 



rs 
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MKDLn;E -JSSR RESPOMSE FORM 



:omplete this question ONLY if searcn results (citation 
print-out) .-^ro no longer available to count the nxjmber 
of relevant citations. If the print-out is available 
GO TO QUESTION 

Mv estimate of the percent of relevant (related to mv 
search question) citations retrieved in my search is: 







OZ 


1 




-J 


1 1 




10 z 






30% 


31 










5 or. 


51 




bOZ 


ol 




70.". 


71 




so* 


31 
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1002 



(GO TO QUESTION 3) 



V >ur searcn retrieved citations* Review eacn ci:ation in 

vour r^rinc-^ut md ccunt the total number of relevant citations, 
relevant citations are ALL citations which you consiaer to be 
r,^lated to vour search question, thus, relevant citations include 

v^se ;>;ith vhica vou vere already familiar prior to the search as 
well as new citations related to vour question. 

Numoer of relevant 

Citat ons in the orint-out 



<G0 TO QUESTION 3) 

Indicate ^^our ie^^ree o: .^at israction with the proportion of relevant 
citations retrieved in tne search bv r^arking an X on the appropri- 
ate aipace belov* 

'Unsatisfactory : : t : : : :^ : Satisfactory 

I : J ^ 5 o T S 9 10 

* COMPLETE ALL rOlLOVrtNG OUESTIONS) 
79 



ERJC tj-. 



N£ I'SER RESPONSE FORM 



RECALL 

Indicate your lewl of agraenent with the following statements by 
•narking an X in cne appropriate space* 

t believe chat the ^iear^h retrieved mo;::t of the relevant 
citations in the MEDLIME data base. 

^ : : t : : :^ : :^ : Agree 



123456789 10 

2* 111 assessing -ny search r^^sults I aia concerned because there 
is no vav to judqe comol-^teness . 

Di^^a^ree : : : : :^ : : : A^ree 

I J 3 - 3 0 3 3 10 

I am concerned beca::so the search results omitted relevant 
cltvAtion.^ with whicn I was familiar prior to tne search. 

jisaw^ree : : : :^ :^ : : :^ :^ ^ Agree 

: : 1 ^ 5 6 : 8 9 10 

r:',e Tiearch results included f-^ver relevant citations than 
I '^xneotec » 

^^i^aaree : :^ :^ :^ : : :^ Aaree 

: : i 5 - : 3 9 iQ 

".insider tne 'Purpose for vhich you requested this searcn. 
Indi.^ate vour vaiie assess^nent of the search results (the 
ibiiitv Che search to meet the need oromptins: vour 
r.^questi bv mar .inu an :i in the appropriate ipace. 

t : : : : : : :^ : Major Value 



: : 3 $ 6 7 3 9 10 

EAaCH INTERVIEW 

ndicatv^ y^ur level of ii^r-^enent with the following statements 
V narking an M in the approoriate space. 

Tne >earcner was knowledgeable concerning the ;se of the 
^lED M'tE lata base for mv question. 

Disagree : : :^ : : : : : : Agree 

123^3^:^? 10 
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medii:;e user respon^se form 



J!. V 'e*:*! that che searcher understood mv requtist after the 
interview. 

Di ia^ree : : : : : :^ : ^: : Agree 

The searcher umnerstood n)y puroose in initiating the request 

Di.:iagree ^: : : : r ^ : Agree 

1234 5 6789 10 

The iearcner vas thorough in exploring all aspects of mv 
searca question. 

Di;ia^ree : : : : : :^ : :^ : Agree 

1 J 5 4 5 6 7 S ) 10 



7he se.irc:ier sui^j^ested terns iporopriate to the subject of 
nv re«iuesi, 

7) is icree ^ : : : :^ : : :^ : ^ Agree 

3 - 5 o 7 8 9 IQ 



r-i' for vocr cooper.\tion i:; completin'G this fcim. 

Do vou wis',', to r^ciiive -i report: .vn cn*^ final project results? 

T es 

N'o 



31 
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PROCEDURES 



^tEDLlNE PROJECT PROCEDURES 



te:c\s medic\l cekter library 



Supplies 

Materialii needtsd for the project include: 

1. Blank cassettes 

2. Tv^lly sheets for MEDLINE Searches not included 

in the project 

3. Consent/Request Forms 

Tape recorder (available at Texas Medical Center Library) 
5. Sample Evaluaton Forrr^ 

Return envelopes for cassettes and Consent /Request fonns 

Population to He Recorded 

For Che purpose of this project it is desirable to seek parti-- 
t^ipation froTn all users who come to tae Library and request a 
MEDl IXE Search, 

Tally Sheets for Mon-Rec:>rdei >gI )L I>-L Searches 

Since 1 would like to know the proportion of recorded >!EDLI^E 
Searches to all MEDLINE Searches conducted durin;? the partici- 
jatiou periods tally r^heets are available to note each MEDLINE 
>^%ircn completed for which there is no recording- On the tally 
sheet please note reason ^ date , and searcher for anv unrecorded 
MEDLINE Search you Miy do. 

Sequence of Events 

I . Consent 

\Tien an Individual requests a MEDLIXE Search, in person, 
explain that the Library is participating in a MEDLINE 
evaluation study and vou are inviting them to participate 
in the project* Give the requester a Consent/Request form 
for ner/his consideration. If the requester agrees, ob- 
tain her/his signature(s) on tne form. fSoirie requesters 
-nay prefer to sign only the first release^ others may sign 
buLii s tatemetits ) * 



^1 
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MEDLIXK PROJECT PROCEDURES 
«ecuence >v . ^vgnts continuea 

3.-4in recoruin.^ m Side ^1 or tha cassette inm:» iiacoiy 
iit^ir the consent ram is ^t^ned* Indicate: 

- Date 
Requ^^ster Name 
Time 

- Search Number (at top of Consent Form) 
Searcher Name 

3 . Re^>uest Fom 

Have user fill out Request Fonr* down to the dotted 
line on page 5* Additional pa^es (d-l) were addeu 
for the convenience or Wavne State University; chey 
s:an be cut off if they are not useful for you) • 

v:!^.eck to see that all information is cotnpleted, 
particularly name. Trailing address and phone number, 

Interview 



Conduct interview accordini^ to usual -nethod 

>j If iaterviev 1,^ lon^^^er than 30 -ninutes , turn 
cas,^ette over :ind continue r^^cording 

^» Alien interview is completed indicate time 
ind >;t >p recording 

:} If interview includes w.:rkinc vith the requester 
It the terT; >al this interaction iihoula be re- 
:ordec: also. 

Atrer Interviv-w 



I) Label cassette vitn 
^:ate 

search number 

re^^iestor name 

^ear::n*-r name or initials 

Check t'> ^ee that Consent rcrm .uas recu»-stv^r -ii^'nature 

'iive tht? requestor a S/iMPLE evaluation form. Indicact:^ 
mat >he/he mav be :on^:xctec in the future to respond 
t 1 >imilir .ut;^. tionnaire . T:;e i^cr i.^ N'tiT tc rc tur:; 
tne ^amrle. it i> for ner ^is inf^-^rrrat ion on>^'* 
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^i!:dl!ne project procedures 



: . After Inverv;ew - continued 

J) Si^n Consent Form in ,searcher .signature position. 
Ir\ for somti reason^ you do not wish to sign even 
though the requestor ha^ agreed to participate, 
It vouid be helpful if vou would note this on the 
signature form so that I will know that it has not 
just been forgotten. 

^ ' After Search 

i) If on-- line citations are printeriv note the riumber at 
citations in the box in the upper right corner of the 
request form. 

^"^i If or f- line citations are sent to the user^ note the 
t.>tal nuinber of off-line citations sent to the user 
MEDLINE ana nay backfiles) * 

If both on*-line and off-line citations result from the 
search note both of tnese numbers in the appropriate 
-jpot in the box in the upoer right of the request forr*, 

'ni^r^ i week T^ail all complected cases ^ the tape plus the 
:.nisent/Req\jest fonn whicn notes the number of citations 
>lv»*n to the tiser) in the pre-addressed envelopes. 



ERIC 



Appendix 



OAKLAND -J N I y H R S ! T Y 



-ear 



.Vs vou recall, you , «>cencly participaced In a trapec interview 
:-r % :-ErL:NE Searcn Request, t^ich an Office of Eiucacioti Grant I a:a 
;cnauit:ir.^ a studv Df user evaluation of on-line searches: you have 
been selected as a MEDLINE user important to this study. I szi 
rscue^tin^ vcur suooort in lor.pietina cha enclosed questionnaire. 
3-ic^ -t i3 anti-ipatec that this stucy will clarify sGtr.e factors 
:.i-.Mvi^ m user satisfaction -^Ith an in-line systea, I hooe vcu vi.^-^ 
ccr.tribvita a few rranutes to this project, the cnnf icentiality :r your 
reol'.' *r;.ll be aaintained, 

:i vou have anv -^uesti^ns icncsrr.ing the for:^, please call 
■ze at tne n'jsber bel:?tj-. 1 anpreccate vour tin:e and cooceraticn, anc 
- ^o---"-' •'i'-.M-arb to receiving vour ccnpletec quest:ionnaire . 



Eileen H:, Hitcru.nghan 
:n~Line Project Dir-ctcr 




Appendix F 



CODINT ;ELIABILir? 
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Coding Reliability 

Several e?rlier studies which have involved coding of 
events occurring in the reference or bibliographic s«,arch 
interview have either siade no specific coder reliability 
determinations (Carsion. 1975) » or have reported reliability 
in terms of simple percent of agreement by coders (Gothberg, 
1975? L^Tich, 1977). Holsti (1969) notes that this method, 
although often used, has been criticized because it fails 
to acco'jnt for the extent of coder agreement arising from 
chance. For this reason a method noted by Waxier and 
Mishler (1966) was utilised in the present study v Cohen's 

is a coefficient of agreement for nominal scales which 
expresses tne proportion of agreement after chance agreement 
has been removed from consideration (Cohen, 1960). 
The formula for computing k: is*. 

V - ^c ^ 
^ N - fc 

where k = the coefficient of agreement 

fo = the sum of agreements observed for each 
category 

fc - the sum of agreements expected by chance for 
each category 
and N = the total number of observations 

An approximation to the standard error of k is given byj 
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'iith a large N (3 100) the sampling distribution of k 
approxiniates normalit:;, so that confidence limits can be 
expressed as; 

95?5 confidence limits = + 1.9^ <f^ 

In the present study (N = 129) a k of .7? is noted {$5% 
confidence interval = .6? — -.S?)* If the more ccmnorJLy 
used percent agreement nethod were applied to the ocserva- 
ticns, the percent of agreement would ecual the upper -.iir.it 
::f the interval (.5?). 
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USER REQUESTS: ANALYZED INTERVIEWS 
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Search Request Statements Presented by Users 
with Analysed Search Interviewc 

I ar. interested in recent literature on reproduction 
in nematodes {an aninial phyla) and nematomorphs. In par- 
ticular the areas of spertaa'wOgenesis, oogenesis, and 
accessory glands in these reproductive processes. Most of 
these studies should cje at the ultrastructural level (i.e.» 
electron nicrcscopy) . 

Cardiac rehabilitation — (exercise training programs) 
relative to the following diagnoses i myocardial infarction, 
angina pectoris, coronary insufficiency. This is to include 
all articles concerning rehabilitative prograins which have 
been established. It is not to be limited by disciplines 
involved. All foreign language articles are to be excluded. 

3lood flow measurement in 1. ovary 2. uterus 3. testis 
^. effect of interuterine devices, and 

The effect of educational prcgrams on incidence of '/D, 
heart disease and breast cancer. (E.g., is the reported 
incidence higher or lower — or sane— after patients or 
public's exposure to ed. of the diseases cr states'?) 

: would like tc find articles about chlorocruorins for 
use in n:y oral exaninaticn and for niy dissertation. 
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Clinical pharmacology of propranolol (uses and indica- 
tions) . 3acl-c to 19?^* 

Proaecx suunmary. To check current literature for 
enlightening inf orTnation. both historical and current, 
relevant to the areas of exercise electrocardiography. This 
especially entails any changes in the electrocardiogram, 
either at rest or during stress, both immediately prior to 
and one nonth following myocardial infarction. Of parlJicular 
interest are changes in 5KG after the subject has undergone 
an exercise program. 

Effect of increased plasma K" on release of 
catechclanines from adrenal medulla* 

The transport of calci'um in r.uscle tissue as it relates 
neuromuscular disease. 

. • ^ «**^v. AT> ^c^iationship cet'A'een apgar 
The subject matter is on --e . ^^.a uo.u-.oi—t' 

scores and length of labor. 

Mii:< composition x age. 

Respiratory rate vs. respiratory failure. 

♦Transport mechanisms and transport of newly synthesized 
proteins into mitochondria. 



From tape} user did not -.vrite cut a statement. 
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Partial denturs design. Effect of occlusal- functional 
forces on P. dentures. Use of precision attachment in P.D. 



Effect on abutments in P.D. 



♦Epitnelial neoplasms. 

Dihydrofolate reductase! the ensyme, its chemistry, 
isolation, etc., and genetics. Also inhibitors, in research, 
not medical uses. 

Health behavior of patients— patient compliance, patient 
attitudes toward health, patient health beliefs. 

Beta adrenergic blocking drugs/exercise. Propanolol. 
nindolcl. Metoproiol. Aebutolol. 
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'From tape; user did net wite out a statement ♦ 
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CODED INTERVIB^ 
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OKt /6/ you would teii n:e what you^d like* /2/ 

I want to ^et a search on beta adrenergic blocking 
drugs # / 0/ 

..♦particularly with reference to exercise /6/ and 
exercise response from the use of the beta adrener§,ic 
blocking drugs . /6/ 

OK, /3/ do you want to list some of those drugs for ne? 
/8/ r don*t know if they all get lumped together. /5/ 

well particularly propranalol. /6/ 

H-A-N /6/ ?ropranalo3 . /6/ Pindolol /6/ 
P-I-M-D-O-L-O-L /6/ 

r.tetoprDlol /V :/l-E-r-O-?-R-O-L-0-L /6/ 
le* Tie see if those are listed together. A/ 



They should be listed '^.der beta adrenergic /o/ or beta 
adrenoreceptor blocking drugs ^/6/ or maybe under 
adrenergic blocking drugs, /6/ 

Ysah /3/ CK, /6/ do you want adrenergic beta reception 



blockaders? 



Yes /3/ that's it. /€/ 



OK, /6/ so you want these. /9/ Let me show you what's 

under there /V so then you can decide whether ycu want 

ther. all of * em cr sose of them. /V 

'j!a hm. /3/ 

OK, /6/ there are those that get listed. /6/ 
hmm. /3/ 

rtnd you can have all of then /c/ rr some of then, .'c/ 
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Si It goes on /6/ 

vear /V Um /iV I probably may as well get, 'un 

;!?**of tnem! /5/ because a lot of papers nave, xike .our 
of them together, /o/ 

St Urn hm. /3/ OK. /3/ 

.-.s -^ard to tell from a title ^vhich ones are left 
ouxr/l/'S.errare four in particular, that I want to ^ 
^tl*' 1- /A/ vmit 've nav as well get tnem all. /o/ x 
;L°-4-'a^ly ;S.-;?o took at the=work that's beer done 

pv3r> — exercise and the use oi these d^ags /-/ 
because there will be many more puolica-ion^ / o/ 
there will be publications with exercise. /&/ 

T.;^-^* /I/ t>ir:ht. /3/ There's usually a lot -written 

0 ^^ow /6/ you're just interested in somecoay 
who has ingested this drug and the e:-ect o. exe.c.se 
on its metabolism? H/ or wna'? A/ 



St 



St 
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•.•e^i -'-e .--^fects of exercise upon blood pressure /6/ and 
i^-se^^/-/ exercise tolerance, /6/ anything liKe that, /o/ 

OK. .3/ 

But 'hey won»t have that in the title. /6/ 

We^l ' ^^st take any» ^rrJ, ^um. incidence of the word 
'exercise appearing with any of tnese? /// 

'^ear. /3/ That should. /5/ that's "^^^^^ J^tjZ^^ 
sh^iid hive, very often, in tne r.ajority oz incidences, 
should contain the word exercise. /:>/ 

Urn hmm /3/ What is the response to exercise? /?/ Is 
it metabolized differently? /?/ or? /^/ 

'Hell no, /lO/ it affects maximum pulse rate /c/ and the 
blood pressure response. /6/ 

OK. /3/ Hew far back do you want to go -ack on this? /S/ 
Urn, well, you nave different categories don't you? A/ 

^ight. /3/ The c>arrent file is January 7? forward /6/ 
but Ihe file itself can go back as far as iVcc. / -/ 



wni» what's the differential in price 'cetween the ...? /7/ 

0K« /6/ the current file is either a $5.00 /6/ or a $7.00 
sear':jh /6/ depending on whether or not you want abstracts 
/6/ ii you go all the way back it's $1S.0C, /6/ or $22.00 
if you want abstracts f /6/ and then in between it rsr*ges 
between those two /6/ so if you want to go bacic to 19^2 
/6/ it's $14.00 /6/ or $16,00 depending on the abstracts. 
/6/ 

let's go back to '72 A/ I have a paper with my account 

CK, /3/ .OK. /3/ And do you want um human and animal 
both? /9/ Are you going to be using...? /3/ 

*Jh» just hunian. /6/ 

you want just English? /?/ or all the languages? 

,'C / 

CK* /^/ Do you have a sense of hew niuch sr^iff is being 
written on this? 



like 50 articles a year? /9/ or Jnore? /9/ or less 



I ir^agins it would be considerable, /5/ it probably 
would be 50 articles a year at least. /5/ 



Maybe more than that. /5/ Some of the drugs have been 
studied much more intensively than others. /€/ 

hmm. /5/ OK. /3/ OK, /6/ and yo^^ affiliation 
is with ...? /^/ 

(gives affiliation} /i/ 



Department? /"/ 
tnedicine. /6/ 

OK. /6/ And you have an account number there? /'^/ 
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OK. /6/ what's a phone number I can reach you at? /?/ 

(gives n^amber) /6/ Now going to be away later in_ 
the wee:<, /6/ but my secretary can come and ge^ i-. 

OK, /6/ This takes about a week or 10 days /6/ unless 
you want the first 10 like tomorrow afternoon. /&/ 

That's CK, /3/ I'm going to be away /6/ so next week 

: ^ / \ / 

OK. /3/ Bid you want abstracts with ix? A/ 
No /*10/ ::o. /lO/ 
wK. /3/ OK. /V 

That would make it -oo numerous like /6/ 

V- name is ^'^ s®"^ ^^'^^ " 

tr.a next day or so. 76/ 



CK. /3/ Hight. /;;/ 

If : have a problem you'll be aro-u-nd tjomorroW? /'^f 
Yes. /3/ till Th^orsday. /c/ 

CK. /6/ v'-^st get back to ycu before then. /6/ 

CK. /3/ 

If it doesn't seem quite... /->/ urv, /"^Z and w.^a.^.-c ^o 
for ... /I/ 

OK, /3/ Thanks. /!/ 
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